search

NASA-PDS / data-upload-manager

Data Upload Manager (DUM) component for managing the interface for data uploads to the Planetary Data Cloud from Data Providers and PDS Nodes.
https://nasa-pds.github.io/data-upload-manager
Apache License 2.0
0 stars 0 forks source link

Improve DUM Upload Performance and Avoid Replication of Files in the Cloud #84

Closed jordanpadams closed 1 month ago

jordanpadams commented 6 months ago

💡 Description

Currently, execution is taking too long to upload data. We need to figure out ways we can improve throughput. TBD on this task. This may be dependent on system CPUs available.

⚔️ Sub-tasks

jordanpadams commented 5 months ago

Removing from release plan for time being until we hear more from SBN-PSI about performance concerns.

viviant100 commented 5 months ago

SBN-PSI noted that --num-threads flag did improve the performance considerably.

jordanpadams commented 5 months ago

📆 04/2024 status: In work. On schedule.

jordanpadams commented 4 months ago

📆 05/2024 status: In work. Delayed 1 sprint due to delays in https://github.com/NASA-PDS/registry/issues/185. No impact on build expected

jordanpadams commented 3 months ago

📆 06/2024 status: In worked. Delayed an additional sprint because blocked by completion of https://github.com/NASA-PDS/registry/issues/185. Will be done by build.

jordanpadams commented 1 month ago

📆 07/2024 status: In worked. Delayed an additional sprint, blocked by completion of https://github.com/NASA-PDS/registry/issues/185. Will be done by build.

jordanpadams commented 1 month ago

Closing for now. Will have second iteration on this task to have registry in the loop

jordanpadams commented 1 month ago

Follow-on here: https://github.com/NASA-PDS/data-upload-manager/issues/151