search

kr / s3

Go package for Amazon’s S3 API
http://godoc.org/github.com/kr/s3
MIT License
107 stars 34 forks source link

Multipart upload of empty file fails #30

Open sqs opened 9 years ago

sqs commented 9 years ago

If you call Create and then immediately Close the file without writing anything (e.g., for an empty file), the upload fails and no file is created. This is because the Close method sends a CompleteMultipartUpload tag with no children:

POST /xxx/asdf.txt?uploadId=xxx HTTP/1.1
Host: xxx
User-Agent: Go 1.1 package http
Content-Length: 51
Authorization: AWS xxx
Date: Wed, 07 Jan 2015 16:57:26 GMT
Accept-Encoding: gzip

<CompleteMultipartUpload></CompleteMultipartUpload>

HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 07 Jan 2015 16:57:26 GMT
Server: AmazonS3
X-Amz-Id-2: xxx
X-Amz-Request-Id: xxx

118
<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema</Message><RequestId>xxx</RequestId><HostId>xxx</HostId></Error>

The AWS API docs at http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html say that the CompleteMultipartUpload element requires one or more Parts.

slimsag commented 9 years ago

Okay, I've been looking into this a lot and I see three potential solutions to this:

1 and 2 below both rely on the idea that we can identify at Close time that no parts have been uploaded (which should be trivial).

  1. Put a false (read: p.len == 0) part into the stream using uploader.putPart in hopes that S3 will take that without error.
  2. Abort the upload using uploader.abort and then upload a zero-byte object using PUT directly (see here).
  3. Only initiate the upload upon the first Write and if we reach Close without any previous calls to Write, upload a zero-byte object using PUT directly (see here)
    • This is by far the most perfect solution, as it wouldn't be using what is probably a corner-case in the S3 API, and it would send no overhead HTTP requests.

I'll see if I can't get access to an S3 account sometime soon to test these potential solutions and make a small example to reproduce the issue.

sqs commented 9 years ago

I implemented @slimsag's fix 2 pretty roughly, but it seems to be working. Check https://github.com/sqs/s3/commit/49acf3c4a7a4b5fe2dd24f636de56c7497430b30.