If you delete the job log directory for an active task, Cylc will preserve its last known status indefinitely. I.e, Cylc will consider the job to be submitted/running forever.
In this case it was caused by a housekeep task being triggered whilst other tasks in the cycle were still running. The housekeep task tarred up the log/job/<cycle> dir removing the job status files in the process.
This situation should be handled similarly to the job no longer appearing in the queue, i.e, the job is dead, long live the job. Stick it into the failed/submit-failed state as appropriate.
Spotted in the wild!
If you delete the job log directory for an active task, Cylc will preserve its last known status indefinitely. I.e, Cylc will consider the job to be submitted/running forever.
In this case it was caused by a housekeep task being triggered whilst other tasks in the cycle were still running. The housekeep task tarred up the
log/job/<cycle>dir removing the job status files in the process.This situation should be handled similarly to the job no longer appearing in the queue, i.e, the job is dead, long live the job. Stick it into the failed/submit-failed state as appropriate.