search

kitware-resonant / django-s3-file-field

A Django library for uploading files directly to AWS S3 or MinIO Storage from HTTP clients.
Apache License 2.0
36 stars 6 forks source link

Clients should handle multipart completion request failure #209

Open zachmullen opened 3 years ago

zachmullen commented 3 years ago

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html

Processing of a Complete Multipart Upload request could take several minutes to complete. After Amazon S3 begins processing the request, it sends an HTTP response header that specifies a 200 OK response. While processing is in progress, Amazon S3 periodically sends white space characters to keep the connection from timing out. Because a request could fail after the initial 200 OK response has been sent, it is important that you check the response body to determine whether the request succeeded.

Our TS client (and possibly other clients, I haven't checked) do not handle this failure mode. Further down on the linked page, it shows an example of what a HTTP 200 failure response looks like:

HTTP/1.1 200 OK
x-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==
x-amz-request-id: 656c76696e6727732072657175657374
Date:  Mon, 1 Nov 2010 20:34:56 GMT
Connection: close
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>

<Error>
  <Code>InternalError</Code>
  <Message>We encountered an internal error. Please try again.</Message>
  <RequestId>656c76696e6727732072657175657374</RequestId>
  <HostId>Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==</HostId>
</Error>
zachmullen commented 3 years ago

Also, I have no idea whether minio has identical behavior.

brianhelba commented 3 years ago

We should also investigate the behavior when CompleteMultipartUpload is called repeatedly (which could mistakenly happen due to retries).