Submitting queued jobs may not be robust

[g-kimbell]

• At least in 0.2.x, I cannot easily tell if this applies to 1.0

• We queue several jobs for one sample on one pipeline

• The Daemon iterates through sorted jobs, checking pipeline statuses in each iteration

• If the pipeline changes status while iterating through jobs, the wrong job can be submitted

[PeterKraus]

Yes, this was a suspicion in 0.2.x. I am not sure where exactly the race condition was, though.

In 1.0, we avoid this by pulling the information about the daemon status only once per iteration, and then acting on this copy for the rest of the rest of the loop: https://github.com/dgbowl/tomato/blob/0a6c4d70185f193d92fb9dbe165b1c6cebbfb223/src/tomato/daemon/job.py#L245-L249