search

ludocode / mpack

MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]
MIT License
533 stars 82 forks source link

Question: Should I use the Expect API for storing filesystem metadata? #50

Closed Ironlenny closed 5 years ago

Ironlenny commented 7 years ago

I've been looking for a serialization format for the filesystem I'm developing. MPack seems to have the features I'm looking for (small binary format, zero-copy, decent documentation, good speed), but I'm not quite sure on what the difference is between the Expect API and the Node API. It seems like I would want to use the Expect API as I'm dealing with filesystem meta-data which has a fixed schema, and the imperative parser is supposed to be zero-copy I believe. Is my understanding correct?

ludocode commented 7 years ago

Yes, that's correct. The expect API is good for data of a fixed schema where you want maximum performance.

As far as "zero copy" goes, for parsing a contiguous in-memory chunk of MessagePack, technically both the Expect or Node API could be used zero-copy for strings and blobs. The Node API just builds a tree of metadata where strings are pointers into the original data, and you can put the node pool on the stack. The Expect API would still be a lot faster though because you can parse it one pass, while the Node API would inherently be multi-pass (one pass to decode the MessagePack and another for you to pull your application-specific data out of it.)

See these benchmarks for a comparison between the Node and Expect APIs. They're a bit out of date and they're for schemaless data, but they should still be relevant.

Just beware that the Expect API is going to be a fair amount of additional work, and is a bit more error-prone. The eventual goal would be to generate code that uses the Expect API from a schema (sort of like Protocol Buffers), but no such generator exists yet. In the meantime, follow the Expect API documentation, and especially take care in breaking out of loops properly.

ludocode commented 5 years ago

I've added a Reader API guide to support the Expect API guide. Hopefully this helps!