Open bjhardcastle opened 3 months ago
Hi @bjhardcastle
Please note the difference between storage options for AbstractFileSystems and options for the their open()
methods:
S3FileSystem class https://github.com/fsspec/s3fs/blob/efbe1e4c23a06e65b3df6a82f28fc49bab0dbd78/s3fs/core.py#L273-L297
The UPath constructor gathers all keyword arguments under **storage_options
and uses those to instantiate the specific filesystem class.
S3FileSystem._open() method https://github.com/fsspec/s3fs/blob/efbe1e4c23a06e65b3df6a82f28fc49bab0dbd78/s3fs/core.py#L611-L625
If you want to pass specific options down to the filesystem specific AbstractBufferedFile implementation, you would use the following in your case:
import upath
upath.UPath("s3://mybucket/myfile.txt").open(cache_type="first")
If you want to set this on the Filesystem level for s3fs you can do:
import upath
p = upath.UPath("s3://mybucket/myfile.txt", default_cache_type="first")
...
p.open() # will use the default_cache_type
Let me know if that helps! It would be wonderful, if you could tell me how I could improve the text in the README to make this more intuitive. PRs are super welcome too!
Cheers, Andreas :smiley:
Hi Andreas,
Thank you very much for explaining in detail. That of course fixed it!
I don't think it was a problem with the README in this case, but the wording for the open()
method (which I assumed came from pathlib
):
Because it says "as the built-in does", I never would have thought to pass it config for the fsspec-related operations.
One of the reasons I use upath is so I don't need to set-up anything manually, it just handles whatever I throw at it! Now I'm trying to use different configurations I'll refer to the documentation more and let you know if any parts aren't clear.
Cheers, ben
Might be related to #204
I'm trying to use the
cache_type
kwarg for s3 [source], but this causes issues down the line when the file is accessed:upath: 0.2.2 python: 3.11.5