calculate serialize size before serialization and serialize to pre-allocated memory

[alan0526]

I am working on inter process communication base on shared memory and Unix Socket Domain recently, I plan to use glaze Beve to serialize my data. I want calculate the object serialization size then alloc shared memory and write serialization data into it. Is there some existing solution to do this. thanks.

[stephenberry]

Serializing to a pre-allocated buffer versus a dynamic buffer like std::string typically has less than a 5% difference in performance. Furthermore, in inter-process communication you should typically reuse your serialization buffer. Glaze uses a doubling of the buffer capacity if growth is needed, so it is rare that you will have new allocations if you reuse your serialization buffer.

Is there a reason why you can't reuse a std::string (serialization) buffer?

A function to calculate the serialization size could be added, but it would need to iterate through your structures. This would actually add more overhead than simply reusing the std::string buffer. So you might have a tiny performance boost for the very first message you send, but subsequent messages would run slower needing to compute the size beforehand.

[alan0526]

Before if I use string I need to serialize to std::string then write it to shared memory, there is an extra copy. And the reason why I need to size is I want to know what the size of the shared memory to be created.

[stephenberry]

I see. I'll keep this issue alive to add the ability to compute the space required before serialization.

I typically use shared libraries which allow me to share the std::string directly across processes, which allows dynamic buffer sharing and avoids the extra copy. But, I can see how if you are wanting to connect services in other programming languages it is easier to not have to deal with dynamic memory and just compute the size beforehand.

[alan0526]

thanks