sharksforarms / deku

Declarative binary reading and writing: bit-level, symmetric, serialization/deserialization
Apache License 2.0
1.14k stars 55 forks source link

Add `to_slice` for buffer reuse `DekuContainerWrite` trait #459

Closed amin-mir closed 3 months ago

amin-mir commented 4 months ago

I'm wondering if it's possible to add a new method to the DekuContainerWrite trait for serializing to an already allocated buffer. Currently, to_bytes returns a new Vec every time it's called, which doesn't allow for buffer reuse. While we can already achieve this using the to_writer on the DekuWriter trait, it involves several steps:

let buffer = vec![0u8; 2048];
let w = Writer::new(buffer);
data.to_writer(w, ());

In contrast, the postcard crate offers a method for this use case:

pub fn to_slice<'a, 'b, T>(
    value: &'b T,
    buf: &'a mut [u8]
) -> Result<&'a mut [u8]>
where
    T: Serialize + ?Sized,

This method directly serializes to a provided buffer, allowing for buffer reuse and potentially improving performance by reducing allocations.

Could we add a similar method to the DekuContainerWrite trait?

wcampbell0x2a commented 4 months ago

This MR adds to_slice. I didn't include the "When successful, this function returns the slice containing the serialized message" as I don't see the use for deku for this? I could be persuaded, but I'm not sure that works with using the Write trait.

amin-mir commented 4 months ago

Returning either the slice containing the serialized message or a usize indicating the number of bytes written would be really helpful. In my example above, the allocated buffer is let buffer = vec![0u8; 2048]; but the serialized message could take only 7 bytes. This is important to know when we're sending these bytes over the network e.g. using tokio TcpStream:

stream.write_all(buffer[0..n]).await?;
wcampbell0x2a commented 4 months ago

Returning either the slice containing the serialized message or a usize indicating the number of bytes written would be really helpful. In my example above, the allocated buffer is let buffer = vec![0u8; 2048]; but the serialized message could take only 7 bytes. This is important to know when we're sending these bytes over the network e.g. using tokio TcpStream:

stream.write_all(buffer[0..n]).await?;

Agreed. Changed to return the amt of bytes written.