search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.58k stars 226 forks source link

Common mutipart signature auth error can cause multipart uploads to fail #696

Open ggprod opened 7 months ago

ggprod commented 7 months ago

Common multipart uploads fail with an error like: ERROR "cp **** s3://****": MultipartUpload: upload multipart failed upload id: dBmwIX7UhK9IotwKi1j1bM.sInQ7jUngrmjZ3GlmNz1GfLmYZ_4b94ReAgZ4OOMm0wQUfm2DQshNxlCyTBFuzlOfMWKnMLIhEYVB7p5iw39vMcNJn375vBDhWH069xCaa24d8SyktuEyNR5jIt.EOGp9fiucRRtXZ9l80.UnARw- caused by: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: H871REM5J5C1BNN0, host id: FE1FVxCXCymin8yLl1DfjNAyx/yWqS9MwSQ0b7Tcum5xEeoMo/nbpZOjqgEtdnsx2XrDFqrR6/0=

Typically re-uploading the failed object succeeds. It may be good to add retries for this type of error (even though it is a typically non-retry auth error)

I should add this was observed uploading to an s3 bucket with transfer acceleration enabled using a command like this: s5cmd --endpoint-url https://****.s3-accelerate.amazonaws.com --numworkers 512 cp --destination-region eu-central-1 --concurrency 512 **** s3://*****

atamano commented 7 months ago

I'm having the same issue with R2

rtsainear commented 3 months ago

Same issue with R2

fsaavedrap commented 3 months ago

Same issue with Cloudflare R2. Tried to upload about 250 objects, ranging from 2 to 10gb in size each, about 200 fail with this error