search

fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
211 stars 36 forks source link

Implementation of 'local://' URI Scheme Support #150

Closed glesperance closed 5 months ago

glesperance commented 9 months ago

Added support for the local:// URI scheme. Although it's less ubiquitous than file://, this scheme is employed by key data engineering projects like Apache Spark and Ray.io for local file handling.

Including this support broadens our software's versatility and adaptability with these projects.

[1] Spark Example -- https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management:~:text=local%3A%20%2D%20a%20URI%20starting%20with%20local%3A/%20is%20expected%20to%20exist%20as%20a%20local%20file%20on%20each%20worker%20node.%20This%20means%20that%20no%20network%20IO%20will%20be%20incurred%2C%20and%20works%20well%20for%20large%20files/JARs%20that%20are%20pushed%20to%20each%20worker%2C%20or%20shared%20via%20NFS%2C%20GlusterFS%2C%20etc. [2] Ray.io example -- https://docs.ray.io/en/latest/data/api/doc/ray.data.Dataset.write_json.html#:~:text=jsonl%22)%0A%3E%3E%3E-,ds.write_json(%22local%3A///tmp/data%22),-Time%20complexity%3A%20O [3] Genera Github Search for "local://" -- https://github.com/search?q=local://&type=code

Note this requires https://github.com/fsspec/filesystem_spec/pull/1381 to work.

ap-- commented 9 months ago

Thank you for the PR @glesperance :heart:

please run the lint session via nox, and as soon as the fsspec pr is merged and a new fsspec release is out we can merge this.

Cheers, Andreas :smiley:

glesperance commented 8 months ago

@ap-- fsspec commit has been merged. Also added linting commit

ap-- commented 8 months ago

Note: waiting for the next fsspec>2023.9.2 release.

ap-- commented 8 months ago

gcsfs==2023.10.0 was released. Rerunning ci.

ap-- commented 5 months ago

Thank you again for your contribution ❤️