liteserver / binn

Binary Serialization
Apache License 2.0
440 stars 58 forks source link

Array serialization #38

Open funbiscuit opened 3 years ago

funbiscuit commented 3 years ago

I've seen there was an issue about array serialization and it was "solved" by using lists instead. But it is quite inefficient in both space and speed, especially on embedded devices (while this library seems to be the best for embedded). To give an example, I send data (array of 100000 uint16) over network split to packets of about 512 bytes each. If I use lists, I need about 600 packets, while I can fit into 400 if I use blobs and interpret them on the other side. This is a 50% difference. In my opinion this library will benefit a lot by adding support of lists of fixed type (and fixed size of each item). Do you think it will be possible? It may be even a small variation to list interpretation where values will be stored one after another and not storing values of different size or type than first item. Currently I use objects with two keys: t and d. t stores type of each element while d stores blob of an array. And I'm almost fine with it (it doesn't handle endianness but I don't need that). But native support would be a lot better.

kroggen commented 3 years ago

I already thought about this, but currently I have no spare time.

There are many details to decide like storage format, interface, if it should support automatic array increase (push or append instead of just access by position) or only fixed size (informed at creation), if supporting arrays of structs or only basic data types...

It is good to know that you are able to use blobs for this.

Maybe in the future, if there is enough demand for that.