[FEATURE] Migrate from fs to fsspec

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.

https://fugue-tutorials.readthedocs.io/

Apache License 2.0

1.98k stars 94 forks source link

[FEATURE] Migrate from fs to fsspec #488

Closed goodwanghan closed 11 months ago

goodwanghan commented 1 year ago

fsspec is a lot more popular than the package fs (based on pkg downloads).
fsspec is used by pandas and dask
cloud storage extensions of fsspec are up tp date. In contrast fs-s3fs latest release is in 2019...

In addition, there is not very strong coupling with fs in Fugue. We need to remove the usage of triad.FileSystem, replace with fsspec, and also remove fs property from ExecutionEngine.