search

tus / tusd

Reference server implementation in Go of tus: the open protocol for resumable file uploads
https://tus.github.io/tusd
MIT License
2.92k stars 465 forks source link

Getting error about part sizes with Cloudflare R2 bucket #1137

Open thedentist8 opened 1 month ago

thedentist8 commented 1 month ago

I use tusd with Cloudflare R2 bucket as storage. When an upload is interrupted and resumed, I get an error "All non-trailing parts must have the same length". Looks like there is a limitation with the R2 buckets (which is not standard for S3), where they require all parts to be the same size.

My guess is that when the upload is resumed, calcOptimalPartSize calculates a different size for the part. Any suggestions on how to solve this and ensure that the part size will be always the same?

Acconut commented 1 month ago

Yes, tusd might emit different part sizes as this is not prohibited by AWS S3 and also helps to optimize the upload to S3. For example, a 80 MB PATCH request with an configured optimal part size of 50 MB, yield a 50 MB and a 30 MB part upload to S3.

If you want tusd to emit parts to S3 with equal sizes (with only the last part having a different size), you can configure the minimum part size to be the same as the optimal part size. I would recommend a value of 20 - 50MB to start with. These configurations can be done either in the S3Store when you are using tusd programmatically or via the CLI flags.

Let me know if this helps, then we can add this to the documentation :)

thedentist8 commented 1 month ago

Makes sense, thanks!

Related question: if I set both to 20MB, and let's say the upload gets interrupted (due a bad network) after the first 15MB, will those 15MB get uploaded as a ".part" file so it can be re-used after resuming, or will that get completely dropped, and the client will start from 0? 

Acconut commented 1 month ago

will those 15MB get uploaded as a ".part" file so it can be re-used after resuming

Yes, this partial part will be reused when the upload is resumed.