njaard / sonnerie

A simple timeseries database
Other
266 stars 19 forks source link

A way to store a byte array #37

Open rubend056 opened 7 months ago

rubend056 commented 7 months ago

Is there a way to store a byte array of a specific size?

I'm asking because I wanted to implement the ToRecord trait for my own struct Limb:

impl ToRecord for &Limb {
    fn store(&self, buf: &mut std::vec::Vec<u8>) {
        buf.write(&postcard::to_vec::<_, 32>(self).unwrap()).unwrap();
    }
    fn format_char(&self) -> u8 {
        b'L'
    }
    fn size(&self) -> usize {
        32
    }
    fn variable_size(&self) -> bool {
        false
    }
}

but I got a panic on row_format.rs on line 151: panic!("invalid format character '{}'", a);

There seems to be more to extending the formats stored than just implementing ToRecord/FromRecord. Is there an easier way to do this? Or should I just use 's' with base64 encoding?

njaard commented 7 months ago

Sorry about my delay in responding.

The code in row_format.rs is necessary for receiving information from command-line tools.

Anyway, I've added this feature in the byte-arrays branch

Let me know if that works for you and I will merge it.

rubend056 commented 6 months ago

Wow thank you for the work you put into this :)

Since then I've been using 's' with a base64 encoding, it seems you're doing the same but with a different format_char 'B'.

Would your implementation have any performance/size benefits over just using 's' with base64?

njaard commented 6 months ago

I conducted an experiment to see if compressing a base64 string with LZ4 results in the same size as compressing the same data when represented in binary, and it's about 30% more.

So, yes, there's a size benefit of not using base64, and converting to base64 probably has cost as well.