search

IBM / ibm-cos-sdk-js

ibm-cos-sdk-js
Apache License 2.0
38 stars 19 forks source link

When using npm 's3-node-client' library with 'ibm-cos-sdk', get "unable to sync: Error: Non-file stream objects are not supported with SigV4 at Object.computeContentMD5 (/Users/timfries/repos/s3test/node_modules/ibm-cos-sdk/lib/util.js:748:23)" #48

Closed timisonyourside closed 5 years ago

timisonyourside commented 5 years ago

npm 's3-node-client' library has a nice feature where you can sync a local directory with an s3 directory. According to it's Github repo at https://www.npmjs.com/package/s3-node-client, you can "Create a client from existing AWS.S3 object."

Tried to do so using the 'ibm-cos-sdk' library, but got the error message in the title indicating that "Non-file stream objects are not supported with SigV4". However, doing some debugging 'ibm-cos-sdk' library indicates that the cos library sets the signature to 'iam', not 'sigv4'.

This appears to be a bug where a multipart upload with an 'iam' signature is not addressed.

I've attached a Node 4 project where I follow the npm 's3-node-client' library instructions to "Create a client from existing AWS.S3 object" which was created using the cos library. Then, I tried to sync a directory with multiple small and large files (one is 700+MB) to s3, and got the "not supported with SigV4" error from the computeContentMD5 source file. Here is a zip of the project: Archive.zip

I am on MacOS running this attached Node.js project from the command line. Please help, thanks!

runnerpaul commented 5 years ago

@timisonyourside we're looking into this.

barry-hueston commented 5 years ago

Internal reference: CSAFE-57052

timisonyourside commented 5 years ago

Any updates?

barry-hueston commented 5 years ago

Hi @timisonyourside, I'll have a response back to you later today on this issue.

timisonyourside commented 5 years ago

Any updates?

kellerbr-ibm commented 5 years ago

So after some digging it looks like s3-node-client is trying to take over the responsibilities of generating MD5 checksums by splitting the file ahead of time and uploading each piece individually through its own wrapper object. The issue is that it delays attaching the "body" of the object while the SDK is trying to determine which headers go on the object, and thus the SDK (which also wants to build the MD5) can't build the MD5 because there's no body yet. This means it also triggers even if you use v4-style credentials with the SDK.

I can discuss changing how that functionality works in the future, although we require it for some resiliency features that we offer on our object storage, so I'm not sure that will change.

For now, if all you need is a simple folder upload and want to continue using that library, I suggest using the AWS SDK and generating some legacy HMAC credentials (accessKeyId/secretAccessKey) that will work with it. You can then set the endpoint in the S3 client instance to the IBM Cloud Object Storage endpoint and it should work without any further issues.

timisonyourside commented 5 years ago

Thanks, @kellerbr-ibm. 1) My goal was to use the IBM SDK to access IBM COS since AWS probably won't offer any support accessing IBM COS if something breaks. Sounds like I can't do that after all. 2) What is the real impact of using the AWS SDK instead of the IBM SDK since the IBM SDK has the resiliency features you mentioned? Are the resiliency features a big deal or do they just come into play in a small percentage of situations. 3) Do you happen to be aware of an open source or IBM package that does s3 folder sync similar to the s3-node-client but without messing up the IBM SDK logic so I can use the IBM SDK? Thanks!

kellerbr-ibm commented 5 years ago

Hey @timisonyourside, sorry for the delay.

If you aren't using advanced features such as server-side encryption, storage tiering, or using the third-party client to set ACLs (beyond public-read) or IAM policies then the AWS SDK should be sufficient. Resiliency features include object retention for government or legal compliance, which prevent anyone (including IBM) from deleting objects for a user-specified period of time for any reason.

The AWS SDK does and will continue to work for standard bucket and object creation, copying, and deletion.

Unfortunately I don't believe we support any third-party clients at this time. If you're familiar with bash, the IBM COS CLI might be easy enough to make some simple scripts with that fulfill your need.

However, we've added support for that client (among others) to our roadmap though I can't say exactly when we'll get to it.

paul-carron commented 5 years ago

@timisonyourside did @kellerbr-ibm response help or do you need more informaiton?