search

DataBiosphere / dsub

Open-source command-line tool to run batch computing tasks and workflows on backend services such as Google Cloud.
Apache License 2.0
264 stars 44 forks source link

Support a new display status to distinguish "QUEUED" vs. "RUNNING" #204

Open mbookman opened 4 years ago

mbookman commented 4 years ago

For historical reasons (initial support from the Pipelines API v1alpha2), dstat provides no distinction between tasks that are QUEUED vs. actually RUNNING. Today they are all listed as RUNNING.

We should surface this distinction in places where the status is displayed. It can be misleading to new users when they submit a large number of tasks, it may appear that some are taking a very long time to run when they are actually queued and blocked by other running tasks that are consuming Compute Engine Quota (such as CPU, Persistent Disk, or In-Use IP Addresses).

mbookman commented 4 years ago

Until this is surfaced in dstat output, users can use one of:

to see the number of concurrently running VMs.

sjgosai commented 1 year ago

Clarification question: Are jobs that are QUEUED due to resource quotas, but listed as RUNNING (i.e. Status: VM starting (awaiting worker checkin)) constrained by the same timeout restriction as jobs that are actually running?

wnojopra commented 1 year ago

Clarification question: Are jobs that are QUEUED due to resource quotas, but listed as RUNNING (i.e. Status: VM starting (awaiting worker checkin)) constrained by the same timeout restriction as jobs that are actually running?

Yes, the timeout includes the time spent waiting for a worker to be allocated. The default timeout is seven days which can be changed with the --timeout flag. See the provider specific parameters section in provider docs