humanmade / Cavalcade-Runner

Daemon for Cavalcade, a scalable WordPress jobs system.
https://engineering.hmn.md/projects/cavalcade/
54 stars 26 forks source link

Allow to wait some time before timeouting initial connection to the database #34

Open onnimonni opened 7 years ago

onnimonni commented 7 years ago

I'm running this in local docker environment and I'm hitting an error which says that database refuses connection:

web_1       | worker.1  | Error: SQLSTATE[HY000] [2002] Connection refused
web_1       | worker.1  | #0 /var/www/project/vendor/humanmade/cavalcade-runner/lib/Runner.php(200): PDO->__construct('mysql:host=mysq...', 'web', 'password', Array)
web_1       | worker.1  | #1 /var/www/project/vendor/humanmade/cavalcade-runner/lib/Runner.php(89): HM\Cavalcade\Runner\Runner->connect_to_db()
web_1       | worker.1  | #2 /var/www/project/vendor/humanmade/cavalcade-runner/bin/cavalcade(26): HM\Cavalcade\Runner\Runner->bootstrap('/var/www/projec...')
web_1       | worker.1  | #3 {main}

This happens because my mysql is not ready yet. Could we add a default timeout which could be adjusted with an flag like --timeout=10?

rmccue commented 7 years ago

Why not build the timeout into your script running Cavalcade Runner instead? (sleep 10 && cavalcade) It seems like a system-specific thing if your MySQL instance hasn't yet started, rather than something specific to Cavalcade.

onnimonni commented 7 years ago

I think it would be better to have it in cavalcade itself so it can also automatically retry few times before failing. There must be other environments where this is preferable default behaviour too.

nathanielks commented 6 years ago

I agree with @rmccue here. If mysql isn't ready yet, then your scheduler or your other orchestration/infrastructure management tools should take into account that a dependent resource isn't available before starting the container, or work it out that the container fails and the scheduler restarts it. Hopefully you've worked out a workaround!