search

warelab / sciapps

SciApps: a cloud-based platform for reproducible bioinformatics workflows
https://www.sciapps.org
Apache License 2.0
2 stars 1 forks source link

Handling BLOCKED jobs #110

Closed liyawang closed 4 years ago

liyawang commented 4 years ago

Testing with the MCrna data, jobs will be blocked when resources are used up.

If there are jobs that are already blocked. The new workflow submitted will generate a workflow id (4d99d834-67fe-4eb3-b43e-ec8fca043218) but with empty content.

The workflow has two jobs submitted to AGAVE but none of them will be written into the SciApps database. Agave Job IDs are 0df94c12-90a2-4ddf-bbd5-f093db8aa904-007 and d06a87f1-3af0-4dee-84ee-be95afde9b2c-007.

liyawang commented 4 years ago

The same error occurred again when the user is running a different workflow. It seems that when there are jobs returned as 'BLOCKED' from the API. SciApps couldn't register new jobs/workflows into the database.

liyawang commented 4 years ago

The issue seems to be on the TACC side. Once the queue is BLOCKED. New jobs submitted to the queue will be ignored (mostly). Thus the workflow will never start since we only try three times on submitting the job. Need to contact TACC on how the BLOCKED jobs are handled.

liyawang commented 4 years ago

Fixed