`path.iterdir()` yields `path` itself as the first item (with S3)

fsspec / universal_pathlib

pathlib api extended to use fsspec backends

MIT License

210 stars 36 forks source link

`path.iterdir()` yields `path` itself as the first item (with S3) #226

Open danielgafni opened 1 week ago

danielgafni commented 1 week ago

Given the following objects on S3:

s3://my-bucket/my-directory/0.txt
s3://my-bucket/my-directory/1.txt

UPath("s3://my-bucket/my-directory").iterdir() yields:

s3://my-bucket/my-directory
s3://my-bucket/my-directory/0.txt
s3://my-bucket/my-directory/1.txt

The first item is wrong, right?

ap-- commented 1 week ago

Hi @danielgafni

Thank you for reporting! Could you check 3 things:

did this change with recent fsspec versions? (Just try installing the newest vs installing one from <2024)
if you list the contents of the bucket just using filesystem_spec (without upath) does it return an entry named my-bucket/my-directory ?
was this s3 bucket manually modified using the AWS webui ? (i.e. Files uploaded via the webui?)

Cheers, Andreas 😊

danielgafni commented 1 week ago

Hey!

I can answer to (1) and (3) right away:

My fsspec version was 2024.3.1
No, it wasn't

Regarding (2), I will be able to check a bit later

danielgafni commented 1 week ago

Checked it, seems to be an issue with fsspec. It has this problem too.

ap-- commented 1 week ago

I wouldn't call it a problem. There's something stored under that key in the bucket. We should just define / document behavior of upath in cases like these.

danielgafni commented 1 week ago

No object with this key exists in the bucket. Also, it happens with any "directory" (common path prefix) in the bucket, not just a specific one.

Also, aws s3 ls doesn't contain the problematic key

ap-- commented 1 week ago

Could you create a PR with a test case in the upath S3 tests that reproduces the issue? That would be super helpful for finding a solution