yummyml / yummy

Apache License 2.0
33 stars 3 forks source link

Cleaner and faster dask backend for delta lake read #33

Open raviranak opened 1 year ago

raviranak commented 1 year ago

Saw the panda conversion in delta read with Dask Backend, so made changes to work with to_pyarrow_dataset
https://github.com/raviranak/yummy-rdev

Screenshot 2023-07-19 at 6 16 08 PM Screenshot 2023-07-19 at 6 15 58 PM
ion-elgreco commented 1 year ago

@raviranak if all you're doing is grabbing the filenames it's easier to do: DeltaTable().file_uris()