fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
211 stars 36 forks source link

S3Path.iterdir() gives wrong paths when used with trailing slash #146

Closed theogaraj closed 9 months ago

theogaraj commented 9 months ago

Which operating system and Python version are you using? Windows 11, Python 3.9.6

Which version of this project are you using? 0.1.3

What did you do?

What did you expect to see? Correct paths of each of the files

What did you see instead? Extra slash in the paths. I get the expected result:

>>> # scenario 1
>>> spath = UPath('s3://tj.test.bucket/firstsubfolder/')  # note trailing slash
>>> for f in spath.iterdir():
...     print(f)
...
s3://tj.test.bucket/firstsubfolder//
s3://tj.test.bucket/firstsubfolder//file_1.txt
s3://tj.test.bucket/firstsubfolder//file_2.txt
s3://tj.test.bucket/firstsubfolder//file_3.txt
>>>
>>> # scenario 2 - same spath but using filesystem.ls
>>> for f in spath.fs.ls(str(spath)):
...     print(f)
...
tj.test.bucket/firstsubfolder/
tj.test.bucket/firstsubfolder/file_1.txt
tj.test.bucket/firstsubfolder/file_2.txt
tj.test.bucket/firstsubfolder/file_3.txt
>>>
>>> # scenario 3 - no trailing slash
>>> spath = UPath('s3://tj.test.bucket/firstsubfolder')  # note NO trailing slash
>>> for f in spath.iterdir():
...     print(f)
...
s3://tj.test.bucket/firstsubfolder/
s3://tj.test.bucket/firstsubfolder/file_1.txt
s3://tj.test.bucket/firstsubfolder/file_2.txt
s3://tj.test.bucket/firstsubfolder/file_3.txt
ap-- commented 9 months ago

Hi @theogaraj,

Thank you for reporting the issue! I can reproduce it locally. Will need to think a bit about how to fix this together with #144.

Cheers, Andreas