search

lemire / streamvbyte

Fast integer compression in C using the StreamVByte codec
Apache License 2.0
374 stars 37 forks source link

Fixes and verifies issue 42 #44

Closed lemire closed 2 years ago

lemire commented 2 years ago

We decoding, our functions may read up to 16 extra bytes from the input (beyond the actual compressed data). Thus the users of this library, for safety, should ensure that there is allocated data 16 bytes beyond the compressed data.

fzhedu commented 1 year ago

@lemire I think we should pad 32 bytes for AVX as the vector lane of AVX is up to 256bits. streamvbyte

lemire commented 1 year ago

@fzhedu We only read into 16-byte registers (so not 32 bytes). However, there may yet still be a bug. Can you share the input that was used to produce this case so we can examine it and produce a fix?

lemire commented 1 year ago

It would be most useful if you can raise this as a new issue.

lemire commented 1 year ago

I do refer you to documentation, this bit is important...

During decoding, the library may read up to STREAMVBYTE_PADDING extra bytes from the input buffer (these bytes are read but never used).

This requirement could be removed.

fzhedu commented 1 year ago

@fzhedu We only read into 16-byte registers (so not 32 bytes). However, there may yet still be a bug. Can you share the input that was used to produce this case so we can examine it and produce a fix?

ok, I will recheck and test again.