Support decoding Union column into DenseUnion - Githubissues

datafusion-contrib / datafusion-orc

Implementation of Apache ORC file format use Apache Arrow in-memory format

Apache License 2.0

28 stars 8 forks source link

Support decoding Union column into DenseUnion #78

Open Jefffrey opened 3 months ago

Jefffrey commented 3 months ago

Initial Union support will decode to Sparse Union: https://github.com/datafusion-contrib/datafusion-orc/commit/ee69b91cb2ce4c18ee5148bd541658d362fc577f

Can try give option to prefer decoding to DenseUnion: https://arrow.apache.org/docs/format/Columnar.html#dense-union

Since ORC already encodes data in a dense manner (omitting nulls in the data streams) so might be more efficient? Unknown, just something to keep in mind