hydra-queue-runner: drop broken connections from pool

[Ma27]

Closes #1336

When restarting postgresql, the connections are still reused in hydra-queue-runner causing errors like this

main thread: Lost connection to the database server.
queue monitor: Lost connection to the database server.

and no more builds being processed.

hydra-evaluator doesn't have that issue since it crashes right away. We could let it retry indefinitely as well (see below), but I don't want to change too much.

If the DB is still unreachable 10s later, the process will stop with a non-zero exit code because of a missing DB connection. This however isn't such a big deal because it will be immediately restarted afterwards. With the current configuration, Hydra will never give up, but restart (and retry) infinitely. To me that seems reasonable, i.e. to retry DB connections on a long-running process. If this doesn't work out, the monitoring should fire anyways because the queue fills up, but I'm open to discuss that.

Please note that this isn't reproducible with the DB and the queue runner on the same machine when using services.hydra-dev, because of the Requires= dependency hydra-queue-runner.service -> hydra-init.service -> postgresql.service that causes the queue runner to be restarted on systemctl restart postgresql.

Internally, Hydra uses Nix's pool data structure: it basically has N slots (here DB connections) and whenever a new one is requested, an idle slot is provided or a new one is created (when N slots are active, it'll be waited until one slot is free). The issue in the code here is however that whenever an error is encountered, the slot is released, however the same broken connection will be reused the next time. By using Pool::Handle::markBad, Nix will drop a broken slot. This is now being done when pqxx::broken_connection was caught.

cc @NixOS/infra

[mweinelt]

Deployed on hydra.nixos.org. We'll see on the next database reboot how well this will work.

[Ericson2314]

Thanks @Ma27. This makes sense to me.