Make even otherwise uncaught errors propagate out through dwq

[chrysn]

Now with text:

This plays a part in finding the current quiet murdock failures, hoping to find the next iteration of them without having to resort to runner admins looking into their consoles.

If any "uncaught exception" error occurs, this takes about everything that looks like a current job (carefully peeking at locals because the exception could predate the acquisition of a job, in which case it's only console errors again for lack of a back propagation path) and subjects them to the usual check: If the haven't been NACKed often enough, they're just NACKed once more for retry (rather than, like so far, just staying unacknowledged forever and being sent to more workers, whereupon which the final worker either crashes before evaluating the NACKs or just hands it back unfinished before it's even crashing). If it was NACKed by enough workers (now, eg. by crashing them), the exception set as a result on the job.

As exceptions may also happen somewhere inbetween jobs, this is all not an exact science, so a warning is added above the backtrace that this was assigned to "any of the current jobs".

Security caveat:

This posts arbitrary tracebacks (no values, as Django's debug mode would do) up to the queue as results. I think we're fine with that as no confidential code is running anywhere here.

[chrysn]

Testing procedure that happened on my side:

• Run disque
• Run dwqw -v -j1
• Run dwqc -r https://github.com/RIOT-OS/RIOT/ -c 1234 -v and write "foo" in there
• Observe (by adding prints; no idea where job results would come back) that the NACK is sent alright, and (when changing the NACK counting condition to always take the other branch) that the error is formatted and sent as well.

[chrysn]

Squashed in my late fixup (in tests, errors showed to be ugly in %r and readable with %s); I hope it's usable that way already.