sile / patricia_tree

A memory-efficient patricia tree implementation written in Rust
MIT License
111 stars 17 forks source link

How to serialize a PatriciaMap ? #25

Closed astariul closed 1 year ago

astariul commented 1 year ago

I followed the serde tutorial on how to do serialization, and tried to serialize my PatriciaMap with this code :

let mut map = PatriciaMap::new();
map.insert("foo", 1);
map.insert("bar", 2);
map.insert("baz", 3);

let serialized = serde_json::to_string(&map).unwrap();

let deserialized: PatriciaMap<usize> = serde_json::from_str(&serialized).unwrap();

However it fails at deserialization with the following error :

invalid type: sequence, expected a byte string at line 1 column 2


So how can I properly serialize / deserialize a PatriciaMap ?

astariul commented 1 year ago

I saw from the unit-tests that serialization is done through postcard :

let mut map = PatriciaMap::new();
map.insert("foo", 1);
map.insert("bar", 2);
map.insert("baz", 3);

let bytes = postcard::to_allocvec(&map).unwrap();

let map: PatriciaMap<usize> = postcard::from_bytes(&bytes).unwrap();

This code seems to work, but I'm not sure it's the proper way to serialize / deserialize ? Because the postcard dependency is a dev-dependency.

sile commented 1 year ago

Thank you for sharing this error. It seems undesirable behavior, so I'm going to fix that.

BTW, when serializaing PatriciaMap, the keys are converted into a byte sequence. However, JSON is a text based format and not the best one to represent byte seuences. So, it might be better to choose other serialization library supporting binary format if you could serialize a large PatriciaMap. (I think that postcard is okay. But, I'm not sure too as I'm not familiar with postcard and just used for the testing purpose.)

sile commented 1 year ago

Fixed. The version 0.5.7 can handle the above serde_jsone example without errors.

astariul commented 1 year ago

Awesome ! Thanks for the quick PR :)

However, JSON is a text based format and not the best one to represent byte sequences.

Then do you have a code snippet that shows how to do that with a non-text-based approach ?
Sorry I'm still beginner with Rust, I don't know how serde works and how to make it work with something else than JSON...

sile commented 1 year ago

The above postcard code is an example of binary format serialization. Other binary format serialization libraries are listed in the serde's overview page (https://serde.rs/). Please refer to the libraries official docs for the usage.

What is the best serialization library highly depends on your usecase (serde_json could be the one). So trying / benchmarking several libraries using actual data is good I think.