TTLabs / EvaporateJS

Javascript library for browser to S3 multipart resumable uploads
1.82k stars 206 forks source link

Resuming after PC wakes up from sleep mode #449

Open hajjimurad opened 4 years ago

hajjimurad commented 4 years ago

I use the library to upload quite large files to S3 (10-20 GB) and it's very likely that the PC will go to sleep mode meanwhile. After it wakes up the upload doesn't continue, it's stuck somewhere in the middle. Is it a bug? And if not, what would be the workaround to continue the uploading?

Thanks!

jakubzitny commented 4 years ago

Hi @hajjimurad, thanks for letting us know. From our experience, in order to properly implement the pause/continue, you should probably use [pause](https://github.com/TTLabs/EvaporateJS/wiki/Evaporate.prototype.pause()) and [resume](https://github.com/TTLabs/EvaporateJS/wiki/Evaporate.prototype.resume()] methods. We are able to continue normally in our implementation, we don't upload 10-20G files though. Everything also depends on your bucket/iam settings.

Can you provide repro of your case, or describe the "stuck somewhere" in more detail? Does AWS return any specific errors, is something happening with the progress of Evaporate instance?

hajjimurad commented 4 years ago

Hi Jakub, thanks for replying.

That's how I instantiate the uploader.

    this.evaporateUploader = await Evaporate.create({
      s3Acceleration: true,
      ...
      maxConcurrentParts: 1,
      cloudfront: false,
      computeContentMd5: true,      
      logging: false,
      customAuthMethod: ...
    });

After waking the PC up there is no more http-requests sent, i.e. no more chunks are getting uploaded. And no http-requests failed. But in the console I see the following error:

image

Which is probably caused by this peace of code:

image

Thank you!

jakubzitny commented 4 years ago

That looks reasonable. Would you find some time to create a test case where this can be simulated please?

hajjimurad commented 4 years ago

@jakubzitny, I was preparing the environment to simulate and found out the reason. Basically I was using the customAuthMethod and didn't take care of the exception handling there. When PC goes to sleep and wakes up after there is an exception happening, because of suspended HTTP-request. If this exception is not handled in place (in the customAuthMethod), then the upload process stops with the error I described below. I still think, the library should take also take care of that, but currently I can avoid the error just by handling the exception.

jakubzitny commented 4 years ago

Thanks a lot for the debugging @hajjimurad. Yes, we'll be trying to address this when rewriting this project to v3.