search

KxSystems / arrowkdb

kdb+ integration with Apache Arrow and Parquet
https://code.kx.com/q/interfaces
Apache License 2.0
28 stars 12 forks source link

Separate nulls for nested char arrays and symbols. #29

Open nugend opened 1 year ago

nugend commented 1 year ago

Don’t believe it’s currently possible. Poked in at the current mappings and it seems tractable given the separation of the writing code paths between symbols and nested char arrays.

Would be nice!

nmcdonnell-kx commented 1 year ago

I think it should be possible to have separate null mappings when writing utf8+kdb symbol vs utf8+kdb string. However, when reading back, utf8 are always decoded as strings so would follow the utf8+kdb string null mapping. Although another issue https://github.com/KxSystems/arrowkdb/issues/31 is asking about a special case (dictionary utf8 keys) for decoding utf8s as symbols so may be some overlap.

nugend commented 1 year ago

I just meant writing, really. Reading probably isn't going to be a problem.