fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
251 stars 44 forks source link

Heads up: pathlib._PathBase may be coming soon #114

Open barneygale opened 1 year ago

barneygale commented 1 year ago

I've logged a CPython PR that adds a private pathlib._VirtualPath class:

https://github.com/python/cpython/pull/106337

If/when that lands, it's not much more work to drop the underscore and introduce pathlib.VirtualPath to the world. This would be Python 3.13 at the earliest.

Would it be suitable for use in universal_pathlib? It would be great to hear your feedback, thank you.

ap-- commented 1 year ago

Hi @barneygale

Thank you for the heads up :heart:

I'll provide more detailed comments in the next days. One immediate thought would be, that

from pathlib import Path, _VirtualPath

class UPath(_VirtualPath):
    ...

assert isinstance(UPath, Path)  # will be False

which won't allow users to use UPath instances in code that is guarded via isinstance(p, Path). But I'll think about it more over the weekend and report back.



Currently, I've been working on an update to universal_pathlib based on the 3.12 pathlib, where I introduce:

class PureFSSpecPath(PurePath):  # handles fsspec URI style paths
    _flavour = fsspecpath  # flavour for fsspec
    ...

class UPath(Path):
    __new__ = ... # dispatches to FSSpecUPath if virtual, else PosixUPath or WindowsUPath

class PosixUPath(UPath, PosixPath):  # shim to have localpaths also be instances of UPath
    __slots__ = ()

class WindowsUPath(UPath, WindowsPath):  # shim ...
    __slots__ = ()

class FSSpecUpath(UPath, PureFSSpecUPath):  # base class for virtual (fsspec based) filesystems.
    __new__ = ...  # dispatches to the filesystem specific implementations

But I don't pass all the tests for the implementations yet, which is why it's not ready yet. I've pushed it to https://github.com/ap--/universal_pathlib/tree/pathlib-update, see: upath.core and upath._flavour

Cheers, Andreas :smiley:

ap-- commented 1 year ago

VirtualPath -> _PathBase see: https://discuss.python.org/t/make-pathlib-extensible/3428/134

barneygale commented 10 months ago

Hey - I've published CPython's private PathBase class in a PyPI package: https://pypi.org/project/pathlib-abc/0.1.0/. No docs yet but I should have them ready soon.

If the PyPI package succeeds and matures, I'm hopeful we can make PathBase public in Python itself. I'd really appreciate any feedback you might have :)