When whe local queue is unavailable, the requests pile up in the zmq queue.

[warownia1]

Whenever a job submission is retried, the new message is pushed to the zmq socket. When the local queue server comes back it fetches the messages from all failed attempts and tries to run them. As a result, the local-queue schedules jobs which will never be retrieved and fail on start as the working directory is not set up.

Suggestions:

• For each failed submission send "cancel job" message. When the queue server is launched, fetch all queued messages and remove those which are cancelled
• Use PUB/SUB pattern instead of REQ/REP. However, queue server needs to send back the job id and state.
• Ad the timestamp and timeout to the jobs. If the job is not picked up within timeout (e.g. 1s) it's discarded on arrival.

[warownia1]

The queue may also fill up on the client-side. Consider using the combination of NOBLOCK with send_json and RCVTIMEO with recv_json so that they'll fail with Again: Resource temporarily unavailable instead of blocking forever.