Use gcloud storage cp instead of Python API - Githubissues

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch

7 stars 2 forks source link

Use gcloud storage cp instead of Python API #249

Closed dchaley closed 2 weeks ago

dchaley commented 2 weeks ago

The Python API is ~80% slower than gcloud storage cp (which uses parallelization and other tricks).

This switches most of our I/O to gcloud storage cp. We didn't convert:

model download. (Uses the cached_open library, which we control but didn't feel like doing yet)
visualization. (Writes images, not numpy, need to generalize to byte streams or something)

Fixes #248.

Paired with @WeihaoGe1009