pytorch / torcharrow

High performance model preprocessing library on PyTorch
https://pytorch.org/torcharrow/beta/index.html
BSD 3-Clause "New" or "Revised" License
642 stars 78 forks source link

`from_arrow` with `List` columns #511

Open myzha0 opened 1 year ago

myzha0 commented 1 year ago

Summary: Adds some basic functionality to allow Arrow tables/arrays with List[primitive_type] columns to be converted to a ta.Dataframe.

Implemented by converting the list column to a pylist and wrapping _from_pysequence. Not super efficient, but provides some functionality to unblock these columns.

Tests: Modified previous test case that checked for unsupported type. python -m unittest -v

----------------------------------------------------------------------
Ran 196 tests in 1.108s

OK