jamesmunns / postcard

A no_std + serde compatible message library for Rust
Apache License 2.0
945 stars 92 forks source link

feat: add function for computing the postcard serialized size of a value. #86

Closed dignifiedquire closed 1 year ago

dignifiedquire commented 1 year ago

Closes #46.

Based on #66.

netlify[bot] commented 1 year ago

Deploy Preview for cute-starship-2d9c9b canceled.

Name Link
Latest commit 9573f9fdd42d45c5305692f4e5cf08cd5056a25a
Latest deploy log https://app.netlify.com/sites/cute-starship-2d9c9b/deploys/63d19e9514f9750009c1657c
dignifiedquire commented 1 year ago

@jamesmunns I am wondering if the Sizer serializer could make use of the MAX_SIZE compile time information if it is available. Not quite sure how that could be integrated.

jamesmunns commented 1 year ago

Hey @dignifiedquire, thanks for putting this together! I have some thoughts, let me know what you think:

I think we could avoid all of the code duplication in src/ser/size.rs by instead using a Serialization flavor. Specifically, you could probably instead duplicate https://github.com/jamesmunns/postcard/blob/7d079303f7395aac32c1b203e81b9e59e9569cda/src/ser/flavors.rs#L134-L209, but don't actually write to the buffer, just increment a counter.

I think this will be more maintainable than remembering to keep the main serializer and the size serializer in sync, as flavors have a much simpler API, and I think cover what is needed here.

This MAY not play nice with things like the Cobs flavor, which expect to be able to "reach back" to serialized data for filling in placeholder values, though it looks like the interaction is "write only".

If you're not sure how to do this, lemme know, and I can either help you out, or take this on at a later date.

I don't think the current implementation would work with say, "to_slice" and "to_slice_cobs", because they will have different "on the wire" sizes. You really only can accurately determine the size with the "right" serializer flavor taken into account. Again - I think this should be handled by using a counting serializer, instead of the flavor being Cobs<Slice<'a>> you'd have Cobs<Counting>.

I am wondering if the Sizer serializer could make use of the MAX_SIZE compile time information if it is available.

So, I touch on this in https://github.com/jamesmunns/postcard/issues/46#issuecomment-1403629699, but there are really two things going on here:

The latter will have a larger runtime cost to handle, but using something like u32::MAX_SIZE will give you 5 bytes, while serializing 42u32 will only take one byte. I'd prefer to make the distinction clearly in the API:

I think at least most embedded folks will end up using MAX_SIZE + to_slice to make sure there is enough size to serialize, and determine the actual size "for free" while serializing.

dignifiedquire commented 1 year ago

I think at least most embedded folks will end up using MAX_SIZE + to_slice to make sure there is enough size to serialize, and determine the actual size "for free" while serializing.

The tricky piece for my usage here is that I would love to have the MAX_SIZE + "length of the slices in my structs", because they are not limited in size, but because they are embedded I can't use MAX_SIZE at all currently. I guess I could use the approach you suggested with max_size = 0 annotation, and then manually add the runtime knowledge.

dignifiedquire commented 1 year ago

I think this will be more maintainable than remembering to keep the main serializer and the size serializer in sync, as flavors have a much simpler API, and I think cover what is needed here.

Yes very much, looks like this was pretty straightforward :)

jamesmunns commented 1 year ago

One minor comment, otherwise overall looks good! If you have time to add some doctest examples, it would be appreciated, but we can merge without if you're in a rush.

dignifiedquire commented 1 year ago

will update tomorrow

dignifiedquire commented 1 year ago

added a small doctest