Look into if `gsutil` can speed up upload time back to GCS in RSP Data Exporter service

lsst-epo / rsp-data-exporter

0 stars 0 forks source link

Look into if `gsutil` can speed up upload time back to GCS in RSP Data Exporter service #51

Closed ericdrosas87 closed 6 months ago

ericdrosas87 commented 6 months ago

User story

As the architect of the citSci pipeline, I need investigate if the gsutil PyPI package is capable of multithreading uploads or rysnc to Google Cloud Storage for performance improvement purposes.

Definition of done

I have determined whether or not the gsutil PyPI package is capable of improving performance with uploads in the RSP Data Exporter service.

ericdrosas87 commented 6 months ago

After reviewing the gsutil PyPI package, I found that package is meant to be used exclusively on the command line and doesn't export any modules for importing in a Py script. As such, I cannot make use of the gsutil cp -m or gsutil rsync capabilities I was hoping to and as such I'm closing this ticket as invalid.