search

apache / parquet-format

Apache Parquet Format
https://parquet.apache.org/
Apache License 2.0
1.81k stars 432 forks source link

[Format] Specify VARIABLE_SIZE_LIST Logical type #437

Open rok opened 4 months ago

rok commented 4 months ago

Arrow recently introduced FixedShapeTensor and VariableShapeTensor canonical extension types that use FixedSizeList and StructArray(List, FixedSizeList) as storage respectfully. These are targeted at machine learning and scientific applications that deal with large datasets and would benefit from using Parquet as on disk storage.

If Arrow's List was stored as BYTE_ARRAY we would likely see reduced overhead due to reading and writing definition and repetition levels. See discussion here. It would therefore be beneficial to introduce a VARIABLE_SIZE_LIST logical type to Parquet.

mapleFU commented 4 months ago

I’was busy previously. Sorry for delaying.

I found it a bit hard for Parquet to optimize tensor, maybe the problem is that rep-def levels for tensor / fixed length byte-array. Maybe I could try to fast check the rep/def-levels in this type

rok commented 4 months ago

Yes, I would expect rep/def levels to be significant overhead for tensors currently.