fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
240 stars 42 forks source link

Add storage options and protocol #135

Closed ap-- closed 1 year ago

ap-- commented 1 year ago

This PR adds .storage_options and .protocol to UPath.

The "non-pathlib" interface of UPath is then:

the first three provide a public interface to access a file via fsspec as follows:

pth = UPath(..., **sopts)

from fsspec import filesystem

fs = filesystem(pth.protocol, **pth.storage_options)
with fs.open(pth.path) as f:
    data = f.read()

This closes #129, and closes #91.

additional changes:

extra note regarding local filesystems

# protocol is empty string for non-uri localpaths:
p = UPath("/path/file.txt")
assert isinstance(p, PosixUPath)
assert p.protocol == ""

# and it's "file" for uri localpaths
u = UPath("file:///path/file.txt")
assert isinstance(p, LocalPath)
assert p.protocol == "file"

Both "file" and "" return LocalFileSystem when using fsspec.get_filesystem_class(protocol).