fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
240 stars 41 forks source link

Have relative_to return PosixUPath #215

Open CompRhys opened 5 months ago

CompRhys commented 5 months ago

Close #214

Test that should pass if the local path code worked as expected

ap-- commented 5 months ago

You can prevent some of the failing tests, by editing

https://github.com/fsspec/universal_pathlib/blob/380144c18f291f0f0a15fe8a02bc265233dd594b/setup.cfg#L22-L28

to use: fsspec >=2022.1.0,!=2024.3.1

This is due to a incompatibility of UPath's test suite with the newest fsspec.

bolkedebruin commented 4 months ago

Where are we with this? I would be nice to have this in, because the current behavior is broken :-).

ap-- commented 4 months ago

With the new fsspec release, this is ready to be merged. I'm on holiday until the end of May and will prepare a new release when I'm back 😊

bolkedebruin commented 4 months ago

Great!

bolkedebruin commented 4 months ago

Thinking and testing this a little bit further (from an Airflow perspective, but I don't think that matters). The following does not seem to make sense:

o = ObjectStoragePath("s3://tmp/test")
t = ObjectStoragePath("s3://tmp/test/subdir/bla/bla.txt")

t.relative_to(o).resolve()

prints PosixUPath('/Users/bolke/dev/airflow/subdir/bla/bla.txt')

which doesn't make sense either way. If posix it should resolve to tmp/test/subdir/bla/bla.txt or if s3 (that should make more sense I guess?) to s3://tmp/test/subdir/bla/bla.txt.

Isn't it better to return a PurePath or keep a reference to the original?

bolkedebruin commented 4 months ago

What I am considering is to have

ObjectStoragePath.relative_to return itself (e.g. an ObjectStoragePath) with a storage_option of parent set to the self.

__str__ then reproduces the relative path by checking if parent was set, but all other mechanisms intact.