Open dotis opened 1 year ago
Looks like it failed for 2 days then went back to working. The error message does look related to #53 (InfluxDB outage):
[2023-07-31 13:41:56,351] {subprocess.py:78} INFO - requests.exceptions.ConnectionError: HTTPConnectionPool(host='35.211.75.134', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES&db=fwc_coral_disease (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f196406e820>: Failed to establish a new connection: [Errno 111] Connection refused'))
Closing as resolved by #53
Need to re-open this. DAGs not running and getting bad gateway error w/no time series data for FK. FGB board seems fine.
This "bad gateway" error means that the gateway to InfluxDB isn't working, and that is almost always because the influxDB container has died.
tylarmurray@fknms-dashboard-04:~$ docker container ls -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[...]
1353c06aedc5 influxdb:1.8 "/entrypoint.sh infl…" 6 weeks ago Exited (137) 5 days ago influxdb
[...]
Same as before in #53 there is again nothing in the logs. I can restart this easily whenever it goes down, but we need to identify why the container is crashing so I can stop it. There may be logs within the container somewhere that I can dig into. I will try that next time; for now I am pouring all spare energy into the POSE proposal.
I triggered DAG runs for FKNMS and FGBNMS dashes and now FK is not working, although it did work yesterday. FGB seems fine. I wonder if this is the same docker issue?