search

dchaley / deepcell-imaging

Tools & guidance to scale DeepCell imaging on Google Cloud Batch
8 stars 2 forks source link

Test notebook with 4 GPUs #6

Closed dchaley closed 4 months ago

lynnlangit commented 10 months ago

In GCP / VertexAI, GCP types are associated with GCE instance types, for example A2 instances ALL include 1 or more GPUs. Other instance types allow for configuration of 1 or more of 1 or more GPU types.

Generally there is span of 7x in pricing between lowest and highest end available GPUs so best-fit configuration of instance type, type and quantity of GPU(s) is key to TCO. Also, GPU pricing (for most GPUs) is variable by pricing tier (on-demand, SPOT or reserved) and account type (enterprise accounts often include pricing discounts).

Finally, each GCP project includes a base level of quota for each type of GPU available - this is 0 or 1 GPU of each type per service for new accounts by default. While Google can adjust quotas, it is at their discretion as to when they will do so.

lynnlangit commented 9 months ago

Test on Lynn's GCP project - get project info from Lynn

lynnlangit commented 9 months ago

Did test (reported results in sheet) w/2 and 4 NIVDIA T4 GPUs. Oddly better perf w/2 GPUs of this type (than 4). Other GPUs to test V100 (2x cost of T4s) and possibly TPU v2 Pod (512 cores) 16GB RAM (costs 1.5x T4 GPU)

lynnlangit commented 7 months ago

Suspect that we need to verify the CUDA drivers and configuration on the runtime used - WIP to test...

lynnlangit commented 4 months ago

Don't think we'll need 4 GPUs - testing with 1 or 2 GPUs max - so closing this one