search

IQSS / dataverse

Open source research data repository software
http://dataverse.org
Other
882 stars 494 forks source link

Globus support: optimize the file size lookup when completed Globus uploads are finalized #10977

Open landreev opened 4 weeks ago

landreev commented 4 weeks ago

In the current implementation of the addFiles workflow, a separate API call is made for each individual file when a completed Globus upload is finalized on the Dataverse end. It's being done like that simply because it was inherited from how direct uploads to S3 are handled. There is no real need to keep doing this - we could instead call the Globus /operation/endpoint/.../ls on the entire folder once, when we get the confirmation that the transfer task has been completed, and populate the sizes for all the files in the upload batch.

In the ongoing prod. use case, the depositors are trying transfer entire 8K-file datasets in one batch. Miraculously, this has worked for at least one dataset, but these size lookups proved to be the bottleneck and took an obscene amount of time.

cmbz commented 3 weeks ago

Moved to Sprint Ready due to urgency, @landreev will size it appropriately.