Use pyarrow.fs.LocalFileSystem as per arrow 4.x

uber / petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Apache License 2.0

1.8k stars 284 forks source link

Use pyarrow.fs.LocalFileSystem as per arrow 4.x #698

Open JayjeetAtGithub opened 3 years ago

CLAassistant commented 3 years ago

All committers have signed the CLA.

selitvin commented 3 years ago

Can you please update docs/release-notes and mention this change? How did you test the change? Did you try using a dataset on s3/hdfs?