search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

google-batch provider and preemptible job support #292

Closed jacekzkominek closed 4 months ago

jacekzkominek commented 4 months ago

Hi, I'm using the latest dsub (0.4.12) to submit some GPU jobs through google-batch. The jobs get submitted, scheduled and run just fine, however they are run on standard instances rather than spot even though I specify --preemptible in my dsub call. Any idea why this might be happening? Is the batch provider not passing through the spot flag, or is the Google Batch API not honoring it?

Here's my call, for the record:

dsub --provider google-batch --project {project} --regions us-east1 --machine-type g2-standard-8 --accelerator-type nvidia-l4 --accelerator-count 1 --preemptible --image {image} --boot-disk-size 100 --retries 0 --wait --service-account {service_account} --name {name_str}
mbookman commented 4 months ago

Hi @jacekzkominek !

In the 0.4.12 release, the preemptible flag value is not wired through to the Batch API. It is going to be in the 0.4.13 release. Our target is to push this out next week.

-Matt

jacekzkominek commented 4 months ago

Got it, thank you @mbookman very much for a prompt response, and I'm looking forward to 0.4.13!

wnojopra commented 4 months ago

Hi @jacekzkominek,

We just released v0.4.13 which includes support for preemptible (spot) instances with the google-batch provider.

When you get the chance, can you verify if it resolves your issues?

jacekzkominek commented 4 months ago

@wnojopra , sure thing, just submitted a test job and the GCP Batch web interface shows it scheduled to run on a Spot instance, so I think everything works well now, thank you!