fsspec / s3fs

S3 Filesystem
http://s3fs.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
869 stars 271 forks source link

PermissionError: The request signature we calculated does not match the signature you provided. Check your key and signing method. #758

Open femtotrader opened 1 year ago

femtotrader commented 1 year ago

Hello,

I'm trying s3fs with Synology C2 storage (it seems to be an inexpensive S3 like storage) https://c2.synology.com/ https://c2.synology.com/en-global/object-storage/overview They say "Compatible with Amazon S3 API / Use your existing tools and code as our API is S3-compatible." See https://kb.synology.com/en-global/C2/tutorial/C2_Object_Storage_Compatible_API

I'm doing

In [1]: import s3fs

In [2]: url="https://eu-002.s3.synologyc2.net"

In [3]: key="MyKey"

In [4]: secret="MySecret"

In [5]: s3 = s3fs.core.S3FileSystem(endpoint_url=url, key=key, secret=secret)

In [6]: s3.ls("my-bucket")

but it raises

PermissionError: The request signature we calculated does not match the signature you provided. Check your key and signing method.

I haven't find any other issue https://github.com/fsspec/s3fs/issues?q=is%3Aissue+permissionerror+signature

Any idea what's wrong ?

Kind regards

martindurant commented 1 year ago

I have not see this specific problem before.

Perhaps https://github.com/boto/botocore/issues/2109 ? You can set the signature version botocore uses by passing config options. Otherwise, maybe you can contact C2 directly to figure it ut.

martindurant commented 1 year ago

Authentication mechanism C2 Object Storage only supports AWS Signature Version 4, which provides stronger security because it supports the SHA-256 hash algorithm.

The Version 2 and other methods are currently not supported.

( https://c2.synology.com/en-global/object-storage/overview )

Perhaps set v4 explicitly, although I thought it was the default.

femtotrader commented 1 year ago

As suggested in https://github.com/man-group/ArcticDB/issues/630, I wonder if uri shouldn't be like

uri=f"s3://https://{region}.s3.synologyc2.net"
martindurant commented 1 year ago

That does not look right for s3fs. The endpoint url has already been set for the filesystem, it should only need to deal in buckets and keys.