search

onyx-platform / onyx-amazon-s3

Amazon S3 plugin for Onyx
1 stars 9 forks source link

Race condition in upload #4

Closed jeroenvandijk closed 7 years ago

jeroenvandijk commented 7 years ago

The progress listener is added after the upload has started.

This is problematic when, for instance, you are using the wrong credentials or choose the wrong bucket name. The result is that the batch will never be acked and you won't know why.

A simple solution is to call upload with the progress manager. The harder parter is to get to the exception and learn about the Cause. The listener can be wrapped, but we haven't found a nice way to get the exception without calling waitForCompletion (blocking) on the upload.

lbradstreet commented 7 years ago

AWS's APIs can be pretty painful in this respect. It took me some gymnastics to get to a solution, but I'm there.

My main issue is what to do now that I can detect the exceptions properly. I am hesitant to kill the job for any failed transfers. Even issues like missing buckets could end up being transient.

How would you feel if I just reboot the peer, and let the job continue on, knowing that you will now see errors in the log as you expected?

jeroenvandijk commented 7 years ago

Yes we didn't see a nice solution either. Thanks for looking into this!

I'd expect that failures don't happen frequently if everything is configured/programmed properly. If we can see the error in the logs, we can do a fix and retry. If there is an issue at AWS, we cannot do much and rebooting the peer would cleanup any dirty state I suppose.

ChrisBlom commented 7 years ago

@lbradstreet we noticed in that in https://github.com/onyx-platform/onyx-amazon-s3/commit/fecee7a1529d4d7d37760aee63e8baa465a6cdfaethe progress listener is still added after the upload is started, see #5 for a fix where the progress listener is attached when the upload is started

lbradstreet commented 7 years ago

Merged and released in 0.9.13.1. Note that 0.9.13.1 also includes Server Side Encryption support, in case that interests you.