Closed dchaley closed 3 months ago
Jobs with GPU have not even started, ~24+ hours later 😨
We've seen two errors:
1:
Quota checking process decided to delay scheduling for the job the-second-batch-t-d13a30be-dc77-49cf0 due to inadequate quotas [Quota: SSD_TOTAL_GB, limit: 500, usage: 480, wanted: 30.], next schedule time 2024-04-17 14:32:34.04100913 -0700 PDT m=+411937.323907321.
We don't know where the 480GB usage comes from 🤷 But we did see 500GB quota in the quotas page.
2:
VM in Managed Instance Group meets error: Batch Error: code - CODE_GCE_ZONE_RESOURCE_POOL_EXHAUSTED, description - error count is 5, latest message example: Instance 'the-second-batch-t-d13a30be-dc77-49cf0-group0-0-blhv' creation failed: The zone 'projects/deepcell-401920/zones/us-west1-b' does not have enough resources available to fulfill the request. Try a different zone, or try again later.
Sad!
tips - use n2-standard-x
instances, also try another region us-central1
(?)
Running STANDARD
provisioning, n1-standard-8 machine, w/ T4 GPU:
STATUS_CHANGED 2024-04-30T21:16:14.809370937Z
Job state is set from SCHEDULED to RUNNING for job projects/148281182590/locations/us-central1/jobs/j20240430-gpu-standard-1.
STATUS_CHANGED 2024-04-30T21:08:44.860491667Z
Job state is set from QUEUED to SCHEDULED for job projects/148281182590/locations/us-central1/jobs/j20240430-gpu-standard-1.
did the run complete ok? How long did it take? Using 1 image of what size?
512x512, completed in 38s of runtime. Creation --> completion was ~8min. Successfully wrote to BigQuery too 🎉 cc @WeihaoGe1009
This means we can work on running an arbitrary file in the container, and handle output. (Right now, the python script hardcodes its input & params. In the notebook we'd just change it inline.)
Closing this issue as we've completed benchmarks on cpu + gpu!
Run a benchmark on Google Batch. Need to adjust the code to fetch the machine type.
☐ benchmark with CPU ❌ benchmark with GPU (after >24hrs)