dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch
8 stars 2 forks source link

Use gs_fastcopy #267

Closed dchaley closed 2 months ago

dchaley commented 2 months ago

Now that gs-fastcopy is released 🎉 …let's use it!

This PR replaces our (now-old) version of read/write with the "Very Official" versions from the library.

Along the way, add guard clauses because we need to, to leverage multiprocessing. We now use multiprocessing in the write operation (upload in chunks vs gcloud storage).

Fixes #266