search

uwiger / sext

Sortable Erlang Term Serialization
Apache License 2.0
105 stars 51 forks source link

encode maps keeping keys in order. #30

Open benoitc opened 7 years ago

benoitc commented 7 years ago

to keep maps in order, we first sort it by keys by transforming them as list. Then we process the keys/values in order.

ex:

2> B = sext:encode(#{ a => 1, b=>1 }).
<<17,1,0,0,0,2,12,176,128,8,10,0,0,0,2,12,177,0,8,10,0,0,
  0,2>>
3> B = sext:encode(#{ b => 1, a=>1 }).
<<17,1,0,0,0,2,12,176,128,8,10,0,0,0,2,12,177,0,8,10,0,0,
  0,2>>
uwiger commented 7 years ago

I noticed that there are no quickcheck tests for maps, so I started adding some. This highlighted that some more implementation is needed – not least prefix encoding.

benoitc commented 7 years ago

i can probably add that. Did you already started the work on it?

uwiger commented 7 years ago

I started working on it. I got pretty far, but then – after starting a 1M test run of prop_sort(), discovered that the keysort order in maps differs from the normal term ordering: floats are always considered larger than ints.

Needless to say, this poses a significant challenge for sext. I haven't yet figured out how to solve that in a backwards-compatible way (or even breaking compatibility just for map keys).

uwiger commented 7 years ago

@benoitc could you check out #31 please?

benoitc commented 7 years ago

@uwiger sorry I missed it ... I will later today thanks :)

uwiger commented 7 years ago

Actually, we (Thomas Arts and I) have determined that there is still some work needed. I'll try to get back to it as quickly as I can.