Closed XiangpengHao closed 4 weeks ago
Thanks for finding this @XiangpengHao!
The bug is actually that BitPackedArray::encode
is only supposed to work on unsigned integer inputs.
When compressing signed ints in the Vortex Compressor
, we effectively always do FoR (Frame of Reference, which subtracts the minimum value and converts to unsigned) or ZigZag (which essentially does a wrapping shift left, converting negative signed ints to odd unsigned ints and positive signed ints to even unsigned ints).
Fix is in #930, will get that merged and cut a patch release :)
fix released in 0.11.0
Hi Vortex! I recently played a bit on Vortex <-> Arrow with FastLane encoding.
However, it seems that current implementation automatically convert signed interger to unsigned variant, as shown in the following code: https://github.com/spiraldb/vortex/blob/develop/encodings/fastlanes/src/bitpacking/compress.rs#L33-L40
Here's a minimal reproducible code, it will panic because the
Int64Array
is converted back toUInt64Array
I tried to comment out the
.to_unsigned()
etc, and it seems to work again. I haven't read FastLane paper carefully, can you provide some hints on why to convert it into unsigned values?