58GB upload fails

[laramee]

Hi there!

First off, thanks for the fantastic Shrine ecosystem. I'm working on a video-heavy Rails product and it's been a joy to work with.

I ran into an issue where the final /complete POST request fails on a 58GB file with a 416 InvalidRange error from S3. After doing some research, it looks like chunk sizes need to match on both Uppy and the Ruby S3 client for this to go through.

I guess this was never found because Uppy defaults to 5MB for all file sizes under 50,000 MB and this must not be a frequent use case.

May be similar to this issue on the tus server.

Hope this is helpful!

[janko]

First off, thanks for the fantastic Shrine ecosystem. I'm working on a video-heavy Rails product and it's been a joy to work with.

That's really great to hear! ❤️

I ran into an issue where the final /complete POST request fails on a 58GB file with a 416 InvalidRange error from S3. After doing some research, it looks like chunk sizes need to match on both Uppy and the Ruby S3 client for this to go through.

Hmm, I thought the size of chunks only need to be determined on the client side, which the Uppy S3 Multipart plugin already does looking at the line of code you linked to. What do you think needs to be configured on the "server side" (uppy-s3_multipart)?

416 Invalid Range is an unusual error to get, I think that something else might be wrong. On which S3 request do you get this error?

[laramee]

Hi @janko!

Sorry for the delay - I spent some time looking into this a bit more. We compressed our file to ~42GB and it went through fine (which is what prompted my theory of chunk size not being the same on client and server)

My understanding is that the content-length header for the complete call isn't correct and that's why the S3 API is returning 416. I saw some options for chunk size in the S3 SDK but they all seem related to server-side uploads and not for the "companion" workflow we have here. Could this be a bug in the S3 SDK ?

If you have any pointers on how we could debug this I'm all ears.

Cheers!

[janko]

I recommend turning on aws-sdk-s3 logging, and finding the /complete request to see what's wrong exactly on the HTTP level:

Shrine::Storage::S3.new(
  ...
  http_wire_trace: true,
  logger: Logger.new("aws-sdk-s3.log"),
)

This should log to aws-sdk-s3.log, so after making the upload you should be able to go to the bottom of that file, find the multipart upload complete request, and verify whether it looks correct. 416 Invalid Range sounds like something that should happen when you're using content range headers, which doesn't sound like something the multipart upload complete request would use 🤔 If there was something wrong with previous chunks, I would expect AWS S3 to return a higher-level error with some kind of a description.

[laramee]

Hey @janko,

Just realized I had forgotten to follow up on this. I just tried out a 79GB file with logging turned on and can still reproduce. Here's the log with a few things redacted. Bucket name is 10 characters long in production (so the same length as _REDACTED_).

aws-s3-sdk.log

Let me know if I can provide anything else to help troubleshoot!

[janko]

The log shows that you've created 10,000 parts, and after that uploading additional parts will fail, because 10,000 parts is AWS's limit. Have you found a way to tell Uppy to upload chunks larger than 5MB?

[laramee]

I checked the network log, it's actually uploading 10,000 ~8MB chunks so the calculation looks OK client-side. See this screenshot for where Uppy calculates the chunk size.

[jrochkind]

S3 docs seem to suggest part size has to be 5G? Could be wrong or i could be misreading.

https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

It also says:

There is no size limit on the last part of your multipart upload.

So i guess they're suggesting there's no actual max file size... you just need to make the last part big enough for the entire rest of the file?

I'm not surprised shrine doesn't currently do that! I'm not sure how realistic that approach is, or how much to trust this particular doc page. there may be an effective/practical limit to what you can get into S3? They do advertise supporting up to 5TB; I wonder what tools support that how, it's not entirely clear how you're supposed to upload such a large file.

I'm curious, @laramee, are you able to succesfully upload these files to S3 using other means, say the aws command line client?

[laramee]

My understanding of these docs is that the S3 API is expecting this:

• File size under 50GB: 5MB chunks no matter the file size (except the last chunk, which can be < 5MB)
• File size over 50GB: (file_size / 10000) chunks
• Max chunk size: 5GB, which means a max file size of 5TB

So in my ~80GB example, we're going over the 5MB threshold and doing ~8MB x 10,000 chunks.

I think the issue is that the server-side calls are assuming the minimum 5MB chunk size, even though Uppy is splitting up the file properly.

The CLI defaults to 8MB chunks. Searching through the CLI codebase for this parameter, it doesn't seem to be increased automatically so I think > ~80GB uploads would fail without specifying a higher chunk size.

FYI we are using DO Spaces, but the chunk size limits seem to be the same.

[janko]

It would've been helpful to know this is happening on DO Spaces sooner, as it can have bugs that AWS S3 doesn't. Can you verify if the same error happens with S3 as well?

What I see when I looked closely at the logs, is that the 416 Range Not Satsifiable happens on the complete_multipart_upload service call, but all parts seem to have been passed in (if some parts have been passed in wrong, the error message should be different). It seems that this status code is related to HTTP ranges, which doesn't make sense, because there are no HTTP range headers in play here 🤷

<- "POST /cache/88aba283f5a53191f93c76b00cfd0a5c.mov?uploadId=2~DdU1MPw9f2sgv3DlzCTPwiBDAg9MXOh HTTP/1.1\r\nContent-Type: \r\nAccept-Encoding: \r\nUser-Agent: aws-sdk-ruby3/3.104.3 ruby/2.7.1 x86_64-linux aws-sdk-s3/1.78.0\r\nExpect: 100-continue\r\nHost: _REDACTED_.nyc3.digitaloceanspaces.com\r\nX-Amz-Date: 20200818T160829Z\r\nX-Amz-Content-Sha256: a87ef3132135bedbd48e3ec7be4c9a7eabec38f3f5bfb5bf9149793081a142e1\r\nAuthorization: AWS4-HMAC-SHA256 Credential=_REDACTED_/20200818/us-east-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=_REDACTED_\r\nContent-Length: 1048995\r\nAccept: */*\r\n\r\n"
-> "HTTP/1.1 100 CONTINUE\r\n"
-> "\r\n"
-> "HTTP/1.1 416 Requested Range Not Satisfiable\r\n"
-> "Content-Length: 221\r\n"
-> "x-amz-request-id: tx0000000000000c8460c4b-005f3bfcfd-22072a-nyc3b\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Type: application/xml\r\n"
-> "Date: Tue, 18 Aug 2020 16:08:29 GMT\r\n"
-> "Strict-Transport-Security: max-age=15552000; includeSubDomains; preload\r\n"
-> "\r\n"

There is nothing to act on here, because the error message doesn't make it clear where to go from here. So, until proven otherwise, I consider this to be a bug in DigitalOcean Spaces.

[jrochkind]

I think the next step would be finding ANY tool that can upload those files to DO.

[laramee]

Thanks guys, I apologize for not mentioning DO from the get go. I'm not too familiar with the quirks of S3 compatibility so I assumed it was all implemented the same.

I'll give this a shot with the AWS CLI, and report to DO + close this issue if it turns out to be a bug on their end!

[jrochkind]

On shrine's end it's all implemented the same. But DO obviously has their own implementation on their end, with it's own bugs or places that doesn't match S3's implementation.

[laramee]

I was able to upload my file to DO via the CLI (aws s3 cp...) and via the Forklift Mac app, both without issue. My understanding is that the CLI manages the multipart upload automatically. Anything else I can do to help troubleshoot this ?

[janko]

Thank you 🙏 It would also be helpful if you could try doing the same upload to AWS S3, and see if you get the same error. That way we know whether the error is DO-specific.

[laramee]

Just gave it a shot and it went through. See log below for the same file via S3.

aws-sdk-s3.log

Should I raise this with DO then ?

[jrochkind]

It does seem to be a DO problem then. "Should I raise this with DO then ?" Depending on how much work DO engineers are willing to do to debug, "shrine uppy-s3-multipart produces this error with DO and not S3" might or might not be enough for them to do anything about it. You could try debugging further to understand what's going wrong/differently than S3, but it's definitely kind of a tricky and time consuming thing to isolate to a lower-level reproduction/explanation of where DO is going wrong or differently than S3 -- which is why we unfortunately probably don't have time to do it for you/DO either.

[laramee]

I tried increasing the minimum chunk size on both client and server side and it goes through properly. I raised it to 20MB, which gives us a 200GB threshold before it starts acting up - this is sufficient for our needs.

I'll open a ticket with DO so they're aware of this glitch but we're fine with this workaround if you guys don't want to investigate this further and want to close it.

For future reference, here's how to increase the chunk size:

In the Uppy options on the frontend, for the S3 multipart plugin:

getChunkSize(file) { return Math.max(20 * 1024 * 1024, Math.ceil(file.size / 10000)); }

In the Shrine S3 storage initialization on the backend:

upload_options: { multipart_min_part_size: 20.megabytes }

[jrochkind]

Nice, good find, thanks @laramee!

[laramee]

I just gave this a shot again without my fix above just to see if DO fixed the issue on their end after 1+ year, and it went though without a hitch. Cheers!