Dynamic bitset blocks size should not be configurable and be a single byte

man-group / sparrow

C++20 idiomatic APIs for the Apache Arrow Columnar Format

Apache License 2.0

17 stars 9 forks source link

Dynamic bitset blocks size should not be configurable and be a single byte #184

Open Alex-PLACET opened 3 weeks ago

Alex-PLACET commented 3 weeks ago

Currently, it's possible to choose any size for the blocks in the Dynamic bitset's template argument, but in the apache arrow documentation: https://arrow.apache.org/docs/dev/format/Columnar.html#validity-bitmaps But using a block size greater than a byte will lead to endianess issues

Klaim commented 2 weeks ago

dynamic_bitest itself is a generic type, it's not designed to be specific to arrow. It's usage in the library, for example there, here the block size is 8bit, which matches the spec, isnt it? It can't be changed by the user AFAIK.

Alex-PLACET commented 2 weeks ago

In that case, it requires modifications to works on big endian architectures. I don't know if we really want to add complexity for cases we don't use.