search

quickwit-oss / bitpacking

SIMD algorithms for integer compression via bitpacking. This crate is a port of a C library called simdcomp.
MIT License
273 stars 30 forks source link

Alternative data types for output? #21

Open jamestwebber opened 4 years ago

jamestwebber commented 4 years ago

This crate is super useful, thanks for writing/maintaining it!

I don't know if this is feasible but it would be nice if this crate could output to different types beyond u32. I needed to output to u8 to save space and I ended up rewriting a lot stuff for my purposes (I am only decompressing and always have the same width so I could simplify things).

Having done so, I think that it'd be possible to add another level of macro magic here and output to u8 and u16 for when those types are needed. I'm not too familiar with programming rust macros though, it might create a huge binary if it wasn't done carefully.

Just thought I'd put it out there as a potential enhancement, thanks again for the crate.

fulmicoton commented 4 years ago

This is not useful for tantivy so I won't implement it, but I accept pull request.

The binary size is indeed a problem, but we can put it behind a compilation flag.

jean343 commented 4 years ago

I would like to see support for u64 for large databases.

fulmicoton commented 4 years ago

@jean343 same answer as above unfortunately...

jean343 commented 4 years ago

Thank you for the answer. While I think about it, we are only storing deltas, and most u64 deltas can be stored in u32. So this isn't as big of a limitation as I thought. Thanks again for your awesome library.

jorgecarleitao commented 3 years ago

I will try to take a stab a this for the Bitpack1.

The parquet format uses this format and accepts both u32 and u64.

jorgecarleitao commented 3 years ago

Not very easy in stable rust: the support for const generics is limited and we need it to call simd instructions irrespectively of the type, which requires a constant generic. Alternative is via macros. Will keep digging.