search

bji / libs3

Other
154 stars 151 forks source link

Can't upload file into Oracle Cloud Storage thru their s3 API #65

Open norb62 opened 7 years ago

norb62 commented 7 years ago

I'm planning to access Oracle Cloud Storage thru the S3 API described here: https://docs.oracle.com/en/cloud/iaas/storage-cloud/cssto/using-s3-api-compatible-clients-access-oracle-storage-cloud-service.html

I used curl to retrieve my S3 secret key (MySecretKey).

I'm using libs3 on the client side and test with s3 binary. I set the env variables as follow: export S3_ACCESS_KEYID=Storage-acme_ export S3_SECRET_ACCESSKEY=MySecretKey_ export S3HOSTNAME=acme_.storage.oraclecloud.com

Where acme is my personal accound and MySecretKey has been authorize as decribe in the doc.

I can list the content of a bucket: ./s3 list clibucket I can download objects from it: ./s3 get clibucket/Vol1/test filename=~/test

BUT I consistantly can't upload a file into the bucket:

./s3 put clibucket/Vol1/test filename=~/test doesn't work.

I enable debugging in libs3 by uncommenting: // curl_easy_setopt_safe(CURLOPT_VERBOSE, 1); and //#define SIGNATURE_DEBUG

here is the trace I got:

tracelog.txt

got error 422 Unprocessable Entity.

According to Oracle documentation (https://docs.oracle.com/en/cloud/iaas/storage-cloud/cssto/error-code-reference-oracle-storage-cloud-service.html):

422 Unprocessable Entity Cause The value of the ETag header specified in the upload request doesn’t match the MD5 checksum of the HTTP response. Solution This error may be due to a problem in data transmission. Delete the specified object and try again.

I tried to tweek a bunch of headers like region, Etag, content-length, x-amz-content-sha256, md5, etc. Nothing seems to help. Any hint to get libs3 put to work in this configuration would be greatly appreciated.

bji commented 7 years ago

I think you should submit an issue to Oracle -- libs3 is known to checksum its requests properly and uploads to Amazon S3 has been working for many, many years now without issue. If Oracle is going to claim to support S3, then I think the burden is on them to be compatible. They should analyze the request and see why it's not working and then consider what it is they are doing that is not compatible with Amazon S3 request handling.

Additionally, they should be informed that there is no ETag header in the upload request, so their error message doesn't even make any sense. See this documentation:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

ETag is documented thusly:

"Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value."

ETag is a header that Amazon responds with, it is not included in the upload request. It is up to the client to use the ETag to verify the upload if it desires. It makes no sense for the Oracle server to report that the upload request even had an ETag header.

bji commented 7 years ago

By the way, thank you for the detailed error report and for your attempts to diagnose the issue. If you drive this to some solution with Oracle then that would be much appreciated. Thanks again.

norb62 commented 7 years ago

Hi bji, thanks for the quick answer. I agry 100%, including regarding ETag. Still, I managed to upload to their cloud, using another client they recommanded on their doc (s3cmd) . So I figured out I could try to make it work somehow on tweaking the client side. But I've done as you recommanded and ask Oracle for help. I'll report any outcome here asap.

snaolekar commented 6 years ago

@norb62 I tried s3cmd but couldn't configure it using the documentation provided in above link. I am getting following error Retrying failed request: /?delimiter=/ ([Errno 111] Connection refused) Do we need to configure anything on the server side or only curl provided in the document is sufficient? Some qn:

Please share if you have any document on this, or any suggestion from you would be very helpful. Many Thanks --satyam