Flakey S3 uploads - Githubissues

edgi-govdata-archiving / archivers.space

🗄 Event data management app used at DataRescues

https://www.archivers.space/

GNU Affero General Public License v3.0

6 stars 3 forks source link

Flakey S3 uploads #40

Open maxrothman opened 7 years ago

maxrothman commented 7 years ago

Occasionally when trying to use keys generated by the app to upload large files to S3, you get permission denied errors.

Potential solutions:

behind-the-scenes direct upload from browsers to S3
- the backend passes the browser a temp key when the user presses the download button
cloud-native development (no need to upload if it's all in the cloud)

kmcculloch commented 7 years ago

My first question is whether there's a manual workaround if the user gets a "permission denied" error. Wait a minute and try again? Download a new key?

My second question is whether automation will avoid whatever condition is triggering "permission denied" or whether it will simply push the authentication issue down into code where it's even harder for the user to see what's going on.

Needs research.

maxrothman commented 7 years ago

IIRC we tried waiting a while and also getting new keys. The issue seemed intermittent, though persistent enough that some people ended up just throwing their data on flash drives and giving it to the organizers to import later.

In the meantime, if you haven't done so already, I'd recommend turning on S3 bucket logging so you can review the permission denied cases when they occur.