Open grapefroot opened 1 year ago
+1, this would be super useful!
Is this something that's planned soon? If not, do you all have any pointers on how to implement this?
Seems like a easy hack would be to just make a wrapper to ListColumnCpu that wraps _from_pysequence
here. I assume this requires a data copy though to a native list (e.g., call array.to_pylist()
and pass to_from_pysequence
)? Would you have suggestions otherwise?
thanks @myzha0
Seems like a easy hack would be to just make a wrapper to ListColumnCpu that wraps _from_pysequence here.
Yeah that requires two copies (arrow -> Python objects -> Velox memory buffer). For that approach to unblock, can just do ta.column(array.to_pylist(), _arrowtype_to_dtype(...))
right?
Hi guys! When trying to use ParquetDataFrameLoader I ran across a problem when trying to load parquet file if it has an array field. It looks like it comes down to torcharrow.from_arrow not supporting array columns. But it seems that torcharrow already has support for array columns. Are there any plans to implement it when loading from parquet files or are there any problems which stop this from being implemented?
The error basically looks like this: