search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
265 stars 44 forks source link

Feature request: eventually non-preemptible retries #160

Closed carbocation closed 5 years ago

carbocation commented 5 years ago

I think this is already on the roadmap and I'm not sure whether there is a better place than the Github issues to note this, but having the option to eventually make a job non-preemptible would be a great feature to have. For example, I can get through 99% of my ~2k job run in 20 minutes, but finishing the last 1% can take another 20-60 minutes because of unlucky jobs getting killed 4+ times.

wnojopra commented 5 years ago

Retry support for starting with preemptible and falling back to non-preemptible VMs has been included with release 0.3.3.

Please see documentation on retries under the "--preemptible flag" section.