search

breadlysm / SpeedFlux

Script to periodically run the Speedtest CLI application by Ookla and post results to InfluxDB.
MIT License
112 stars 68 forks source link

Retry backoff #33

Open Selta opened 2 years ago

Selta commented 2 years ago

Currently, when running with an influx DB that starts in another container, the Speedflux will complete its 3 retries before the influxDB is accepting connections (even with depends_on set in the compose yaml). According to logs, it often gets through these retries in roughly 1 second, as exampled here:

2022-10-02 05:38:56,504 [INFO] Speedtest CLI data logger to InfluxDB started...
2022-10-02 05:38:57,515 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,515 [ERROR] Full Error follows

2022-10-02 05:38:57,515 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ff9c8e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,515 [ERROR] Retry 1: Initiliazing DB.
2022-10-02 05:38:57,521 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,522 [ERROR] Full Error follows

2022-10-02 05:38:57,522 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ffbba00>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,522 [ERROR] Retry 2: Initiliazing DB.
2022-10-02 05:38:57,527 [ERROR] Connection to influx host was refused. This likely means that the DB is down or INFLUX_DB_ADDRESS is incorrect. It's currently 'influx-grafana'
2022-10-02 05:38:57,527 [ERROR] Full Error follows

2022-10-02 05:38:57,527 [ERROR] HTTPConnectionPool(host='influx-grafana', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb36ff12670>: Failed to establish a new connection: [Errno 111] Connection refused'))
2022-10-02 05:38:57,527 [ERROR] Retry 3: Initiliazing DB.
2022-10-02 05:38:57,533 [ERROR] Database Init failed for 3rd time. Exiting

This PR adds a simple exponential backoff to those retries, allowing influxDB a little more time (~28seconds with the default 2s backoff_in_seconds param) to start and accept connections. Comments and criticisms welcome, of course.