search

tinylib / msgp

A Go code generator for MessagePack / msgpack.org[Go]
MIT License
1.79k stars 190 forks source link

Decoding non-string-able-keyed maps #232

Open fillest opened 6 years ago

fillest commented 6 years ago

Hello. We have some data that is currently encoded with msgpack/msgpack-python and decoded with vmihailenco/msgpack. I'm trying to switch the decoding to msgp. I've added duplicate decoding code and reflect.DeepEqual the results and diffing them when it fails. Where I'm getting with vmihailenco/msgpack:

 TopicIDs: (map[int]bool) (len=1) {
  (int) 360: (bool) true
 },

with msgp I'm getting TopicIDs: (map[int]bool) <nil> silently. The docs say map must have str or bin keys, is that right? It also says "You can still manually decode arbitrary maps with the primitives built into the library", but how? So:

  1. such decoding silently loses the data, it should at least return an error
  2. how to manually decode such arbitrary maps?
erikdubbelboer commented 6 years ago

What happens when you use TopicIDs map[string]bool in your struct?

There is not really a reason for msgp to not allow other types as keys seeing as encoding/json supports this now as well.

The online editor on https://msgpack.org/ doesn't seem to allow int keys either.

You can manually decode it using an extension. This means msgp will pass a raw []byte to your extension code which then has to determine by itself how to decode this. You can use functions like ReadMapHeaderBytes, ReadMapKeyZC, ReadBoolBytes and ReadIntBytes to parse the data.

Dieterbe commented 4 years ago

Has anyone figured out how to actually do this (marshalling or unmarshalling a map with non-string keys). The wiki page for custom extensions seems geared towards single, custom types. Not sure how to set the struct tags on a field that is a composite type (which in my case is a type alias for the map composed of my custom type as key and integers as values). Are we supposed to write the custom extension for the map as a whole, or only for the key type? (I did it for the key type but it's not getting picked up by the library when running go generate)

fillest commented 4 years ago

I was going to try to (back then), but never-ending tasks with higher priority don't ever end :) And in the meantime, our CPU bottlenecks shifted a lot, so I have no idea when I'll get to it..

Dieterbe commented 4 years ago

BTW I agree

You can still manually decode arbitrary maps with the primitives built into the library.

this could use some expanding. which primitives? custom shims? extensions? on the map level or the key level? can we get an example for both encoding and decoding? thanks.