In working on the ATS backend, I made a mistake with the submission logic such that invalid job requests made it to the scheduler. This should probably be handled, at least partially, by an intervening step, but an error in sched that caused it to skip the job left the job in an un-runnable state without informing the submitter or cleaning it up. Specifically the error was the lack of an "nnodes" field that caused the query_rdesc call to fail, and the jsc_query_jcb call in turn.
In working on the ATS backend, I made a mistake with the submission logic such that invalid job requests made it to the scheduler. This should probably be handled, at least partially, by an intervening step, but an error in sched that caused it to skip the job left the job in an un-runnable state without informing the submitter or cleaning it up. Specifically the error was the lack of an "nnodes" field that caused the query_rdesc call to fail, and the jsc_query_jcb call in turn.