AllenNeuralDynamics / aind-data-transfer-service

FastAPI service to run data compression and transfer jobs on the hpc
MIT License
1 stars 0 forks source link

Prevent duplicate upload jobs #60

Open bjhardcastle opened 9 months ago

bjhardcastle commented 9 months ago

Is your feature request related to a problem? Please describe. By accident, we managed to submit duplicate upload jobs twice this week: once through the website (not sure how this happened) and once via HTTP request (ran the same request twice).

We then weren't able to cancel the duplicate jobs - for one of them we didn't even realize until Jon alerted us.

In the end, running two uploads of the same data simultaneously didn't cause an issue: the second operation on each file must have seen that the data already existed and skipped it. A duplicate run of the sorting capsule was started after upload, and would have been wasteful had Jon not canceled it.

I can't think of a case where someone would want to upload the same session multiple times simultaneously, so I propose that the server could prevent this from happening.

Describe the solution you'd like Before allowing a new upload job to be submitted, check there isn't already an upload job for that session in progress.

If there is a reason to allow multiple uploads with the same session ID, then compare csv/job upload parameters instead.

Describe alternatives you've considered

jtyoung84 commented 8 months ago

@jtyoung84 Convert to User Story