search

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch
8 stars 2 forks source link

Container & other settings have to come from config or params #314

Closed dchaley closed 3 weeks ago

dchaley commented 3 weeks ago

Previously we "configured" the container image by changing the code constant before submitting the job. Gnarly but it worked.

Now, one job submits another: meaning: the segmentation code running in the container submits the next job (measurement).

This means that 2nd container (at least) needs to be configured or passed as a parameter.

The parameter (--container_image my-repo/somewhere/container:tag) is the bluntest, and easiest approach. It retains statelessness which is nice.

At the same time it would be valuable to "configure" the Batch environment. We could: (1) have config files we pull from cloud storage (2) use Cloud Firestore

I'm leaning toward (1) rather than adding a parameter just for the container. We also need the region & other settings like the networking etc config we already have.

dchaley commented 3 weeks ago

Finished at long last–

Image

The feedback cycle on this was rough, it may be worth progressing #225