Benchmark DeepCell on Google Batch with Production Image(s)

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch

7 stars 2 forks source link

Benchmark DeepCell on Google Batch with Production Image(s) #198

Closed lynnlangit closed 4 days ago

lynnlangit commented 3 months ago

Continue baselining using my GCP infra:

[ ] Run with SPOT CPU - note outcome / time to run
[ ] Run with SPOT GPU - use 1 T4 - note outcome / time to run
[ ] Run with SPOT CPU AND GPU - note outcome / time to run
[ ] Run single image with SPOT CPU/GPU - note outcome / time to run
[ ] Run single production sized image with SPOT CPU/GPU - note outcome / time to run
[ ] Run THREE production sized images with SPOT CPU/GPU - note outcome / time to run
[ ] Weihao to run on my infra (LMK when ready, I'll add her permissions)

dchaley commented 3 months ago

Does running on 3 images mean one job with 3 tasks run in parallel? Or submitting 3 jobs with 1 each and seeing what happens.

lynnlangit commented 3 months ago

Does running on 3 images mean one job with 3 tasks run in parallel? Or submitting 3 jobs with 1 each and seeing what happens.

Try running 1 job with 3 tasks with parallelism set to 3. Expected behavior would be to spin up 3 GCE VMs (depends on VM size specified, i.e. n2-standard-8, n2-standard-16.... and number of cores specified)

dchaley commented 3 months ago

Blocked on: #202

dchaley commented 2 months ago

Benchmark result for 140M pixels using spot cpu + gpu. (n1-standard-8 + 1 T4 gpu)

Job creation time: 2024-05-23 1:35:40 PM Job start time: 2024-05-23 1:39:37 PM Job finish time: 2024-05-23 1:46:59 PM Total time: 11min 19s

Time spent in container:

Total time: 349.75s Preprocess time: 34.14 Inference time: 103.84 Postprocess time: 176.05

Total time on Vertex AI: 427s (!!)

Why 80s slower on the vertex notebook?

dchaley commented 4 days ago

This is long since complete, in various forms 📖 → 📓