search

spiraldb / vortex

An extensible, state-of-the-art columnar file format
https://vortex.dev
Apache License 2.0
974 stars 25 forks source link

Support i128/u128 as NativePType #1110

Open a10y opened 3 weeks ago

a10y commented 3 weeks ago

There are a number of suboptimal ways we treat VarBinView compute functions because we cannot support u128 as a native PType for PrimitiveArray

robert3005 commented 3 weeks ago

FWIW I investigated it and the only weird thing is Arrow uses u128 and u256 only for Decimal arrays not for primitive ones. So those don't translate to arrow directly. Arrow workaround is to have a typed buffer that can be u128 but not array

a10y commented 3 weeks ago

Hmm, perhaps then VarBinView should hold the views in a Buffer rather than child array, side-steps that

robert3005 commented 3 weeks ago

You could also do that just have to make sure it's properly aligned. I was thinking you could compress that view buffer but we can deal with it later