RequestTimeTooSkewed error on long-running requests

[michaelkeenan]

I was getting RequestTimeTooSkewed errors from S3 when requests ran longer than a certain time (I think it was ten minutes - ETA: probably fifteen minutes). This turns out to be because a datetime is encoded in the signature for each part, and if it differs too much from S3's server time, S3 no longer accepts the PUTs and instead returns the RequestTimeTooSkewed error. The times get out of sync because the signatures for all the upload's parts are created when the upload is initialized, but the parts get sent sometime later. The solution is to generate the signature just before the part is sent, rather than when the whole upload is initialized.

I've fixed this in my fork, but my fork is too different to make a pull request from, and I'm not very confident that what I've done is ideal. But I'll describe how I fixed it:

In upload.js, in the init function, remove the upload.signPartRequests block

In uploadpart.js, change the activate function to this:

UploadPart.prototype.activate = function() {
  var upload_part = this;
  this.upload.signPartRequest(this.upload.id, this.upload.object_name, this.upload.upload_id, this, function(response) {
    upload_part.xhr.open('PUT', '//'+upload_part.upload.bucket+'.s3.amazonaws.com/'+upload_part.upload.object_name+'?partNumber='+upload_part.num+'&uploadId='+upload_part.upload.upload_id, true);

    upload_part.xhr.setRequestHeader('x-amz-date', response.date);
    upload_part.xhr.setRequestHeader('Authorization', response.authorization);

    upload_part.xhr.send(upload_part.blob);
    upload_part.status = "active";
  });
};

In s3mp.js, in the beginUpload function, remove this if condition (though keep the contents of the block):

if (i[key] === num_parts) {
  ...keep this section...
}

Also in s3mp.js, add this function:

S3MP.prototype.signPartRequest = function(id, object_name, upload_id, part, cb) {
  var content_length, url, body, xhr;

  content_length = part.size;

  url = "/s3_multipart/uploads/"+id;
  body = JSON.stringify({ object_name     : object_name,
                          upload_id       : upload_id,
                          content_length : content_length,
                          part_number : part.num
                        });

  xhr = this.createXhrRequest('PUT', url);
  this.deliverRequest(xhr, body, cb);
};

The vendor/assets/javascripts/s3_multipart/lib.js file is the concatenation of all the Javascript files, so make all the changes there too (or regenerate that file).

[garman]

@michaelkeenan I am experiencing an issue with the gem that I think may be related, but I'd like to ask a question of you, and how this was manifesting itself.

I have users occasionally (approx. 1 file per 1000 or more) experience an issue where the upload will progress as expected, then hang. I have been unable to replicate the issue (even with the same file), so I don't have the browser console output, but my rails logs do not shed any light.

My question: were you experiencing similar effects? i.e. the upload hung? If not, how did this error manifest?

Thanks for your time!

[michaelkeenan]

My users would report a couple of different things - sometimes that the upload would hang, like you said, and sometimes that the upload would appear to complete, but never actually complete.

I suspect that it was hanging in all cases, and that there was an additional problem with how the percentage of progress was calculated. It would overestimate how completed it was, so it would get to 100% (and beyond), and then hang. For some reason, in my tests, it would sometimes need to upload up to 60% more data than the file actually contained. So I changed how the percentage is calculated in my fork, making it a percentage of parts uploaded out of total parts, rather than a percentage of data uploaded out of total data to upload. If you also have this problem, you might want to look at my change here.

(It's not especially well-thought-out code, and it has long comments of me thinking to myself about the problem - it wasn't meant for others to see.)

Though, if you're not getting the percentage calculation problem, then all that was a very long way to say: yes, from the symptoms it sounds like the same issue.

btw you should be able to replicate it if you can upload a file large enough that it'll take so long to upload that the datetimes will get out of sync. This source says it's fifteen minutes.

[garman]

this is suprisingly good news, I too had the same upload percentage issue, so I changed it to reflect parts as well.

I will be borrowing your changes here to hopefully fix this issue for my users.

Thanks!

[joseluistorres]

I was notified of that 100% and beyond issue as well, I'm going to include and test that in our fork https://github.com/praxik/s3_multipart/ Thanks @michaelkeenan @garman

[garman]

@michaelkeenan Have you experienced any issues with this change on Firefox, with CORS failing on some parts of an entire upload?

I'm getting: Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at ...

This only happens some times, only in FF, and on seemingly random parts of an entire upload.

Didn't happen before implementing this change, but we also migrated from self-hosted to EC2 around the same time, so Im not entirely sure this is the issue yet.

Thanks for whatever input you can provide.

[michaelkeenan]

Hi Daniel,

I just checked my logs, and there's no record of that error occurring.

I am still having a few problems, though, with RequestTimeout and IncompleteBody errors. I've been searching for other solutions, and it looks like FineUploader (http://fineuploader.com/) handles multipart uploading, and it's frequently updated, and the author is responsive. I haven't fully committed to the rewrite, but I plan to see if that'll be a better way to solve my uploading troubles later this week.

[michaelkeenan]

I apologize for it probably being really rude to post about another solution in the issues of a gem, but...

I've just finished rewriting my uploading system to use FineUploader (http://fineuploader.com/), and it's pretty great. It seems reliable, upload progress is calculated correctly, the default UI looks good, the documentation is very usable, there are lots of options, the debug mode is very helpful, it's actively maintained, and it seems well supported. The only downside is you have to write a bit of server-side code to sign the requests. They have example code for a lot of server languages, but not Ruby. If anyone wants, you can email me (see my profile) and I'll email you my version of the server signing code.

[rnicholus]

@michaelkeenan Fine Uploader also supports a client-side signing workflow.

[garman]

Well, the two of you have convinced me. Tomorrow begins the rewrite.

[rnicholus]

Full disclosure: I'm the developer of Fine Uploader.