search

kizniche / Mycodo

An environmental monitoring and regulation system
http://kylegabriel.com/projects/
GNU General Public License v3.0
2.95k stars 494 forks source link

Daemon not running with 8.13.9 on two raspberry zero #1191

Closed pe2269 closed 2 years ago

pe2269 commented 2 years ago

Describe the problem/bug

Hi, Since i have updated the new version of mycodo 8.13.9 (from 8.12.9) on two raspberry zero,

The daemon is not running any more., (for one of them the daemon has been working for only one day).

I check the troubleshooting for that issue i have no pb with the upgrade.

When i check if the daemon is running : ps aux | grep '/var/mycodo-root/env/bin/python /var/mycodo-root/mycodo/mycodo_daemon.py'

pi 16745 0.0 0.4 7484 2092 pts/0 S+ 12:55 0:00 grep --color=auto /var/mycodo-root/env/bin/python /var/mycodo-root/mycodo/mycodo_daemon.py

When i check the status of mycodo : systemctl status mycodo

● mycodo.service - Mycodo server Loaded: loaded (/home/pi/Mycodo/install/mycodo.service; enabled; vendor preset: enabled) Active: activating (start-pre) since Sat 2022-05-14 12:27:33 CEST; 30min ago Cntrl PID: 9268 (bash) Tasks: 2 (limit: 877) CGroup: /system.slice/mycodo.service ├─ 9268 /bin/bash /var/mycodo-root/install/influxdb_wait_until_running.sh └─17469 sleep 1

mai 14 12:27:33 raspberrypi systemd[1]: Starting Mycodo server...

And I have no daemon lock file on the directory : /var/lock/mycodo.pid

i have paste the web log below because it's the only thing i found suspicious ... :/

Versions:

Mycoco Web Log :

mai 14 12:37:56 raspberrypi python[377]: self._send_output(message_body, encode_chunked=encode_chunked) mai 14 12:37:56 raspberrypi python[377]: File "/usr/lib/python3.7/http/client.py", line 1030, in _send_output mai 14 12:37:56 raspberrypi python[377]: self.send(msg) mai 14 12:37:56 raspberrypi python[377]: File "/usr/lib/python3.7/http/client.py", line 970, in send mai 14 12:37:56 raspberrypi python[377]: self.connect() mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect mai 14 12:37:56 raspberrypi python[377]: conn = self._new_conn() mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn mai 14 12:37:56 raspberrypi python[377]: self, "Failed to establish a new connection: %s" % e mai 14 12:37:56 raspberrypi python[377]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xb269a3b0>: Failed to establish a new connection: [Errno 111] Connection refused mai 14 12:37:56 raspberrypi python[377]: During handling of the above exception, another exception occurred: mai 14 12:37:56 raspberrypi python[377]: Traceback (most recent call last): mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/adapters.py", line 450, in send mai 14 12:37:56 raspberrypi python[377]: timeout=timeout mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen mai 14 12:37:56 raspberrypi python[377]: method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment mai 14 12:37:56 raspberrypi python[377]: raise MaxRetryError(_pool, url, error or ResponseError(cause)) mai 14 12:37:56 raspberrypi python[377]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SELECT+LAST%28value%29+FROM+bool+WHERE+device_id%3D%272136a0ba-9581-401f-a609-13f78adc2207%27+AND+channel%3D%270%27+AND+measure%3D%27gpio_state%27+AND+time+%3E+now%28%29+-+4s&db=mycodo_db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb269a3b0>: Failed to establish a new connection: [Errno 111] Connection refused')) mai 14 12:37:56 raspberrypi python[377]: During handling of the above exception, another exception occurred: mai 14 12:37:56 raspberrypi python[377]: Traceback (most recent call last): mai 14 12:37:56 raspberrypi python[377]: File "/home/pi/Mycodo/mycodo/mycodo_flask/routes_general.py", line 293, in last_data mai 14 12:37:56 raspberrypi python[377]: value='LAST', past_sec=period) mai 14 12:37:56 raspberrypi python[377]: File "/home/pi/Mycodo/mycodo/utils/influx.py", line 182, in query_string mai 14 12:37:56 raspberrypi python[377]: raw_data = dbcon.query(query).raw mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/influxdb/client.py", line 527, in query mai 14 12:37:56 raspberrypi python[377]: expected_response_code=expected_response_code mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/influxdb/client.py", line 342, in request mai 14 12:37:56 raspberrypi python[377]: timeout=self._timeout mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/sessions.py", line 529, in request mai 14 12:37:56 raspberrypi python[377]: resp = self.send(prep, send_kwargs) mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/sessions.py", line 645, in send mai 14 12:37:56 raspberrypi python[377]: r = adapter.send(request, kwargs) mai 14 12:37:56 raspberrypi python[377]: File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/adapters.py", line 519, in send mai 14 12:37:56 raspberrypi python[377]: raise ConnectionError(e, request=request) mai 14 12:37:56 raspberrypi python[377]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SELECT+LAST%28value%29+FROM+bool+WHERE+device_id%3D%272136a0ba-9581-401f-a609-13f78adc2207%27+AND+channel%3D%270%27+AND+measure%3D%27gpio_state%27+AND+time+%3E+now%28%29+-+4s&db=mycodo_db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb269a3b0>: Failed to establish a new connection: [Errno 111] Connection refused')) mai 14 12:42:14 raspberrypi python[377]: 2022-05-14 12:42:14,064 Pyro5 Communication error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:42:20 raspberrypi python[377]: 2022-05-14 12:42:20,805 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:43:20 raspberrypi python[377]: 2022-05-14 12:43:20,864 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:44:20 raspberrypi python[377]: 2022-05-14 12:44:20,984 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:45:20 raspberrypi python[377]: 2022-05-14 12:45:20,970 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:46:20 raspberrypi python[377]: 2022-05-14 12:46:20,940 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:47:20 raspberrypi python[377]: 2022-05-14 12:47:20,990 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:48:25 raspberrypi python[377]: 2022-05-14 12:48:25,947 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:48:51 raspberrypi python[377]: 2022-05-14 12:48:51,680 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:49:21 raspberrypi python[377]: 2022-05-14 12:49:21,110 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:49:40 raspberrypi python[377]: 2022-05-14 12:49:40,834 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:50:06 raspberrypi python[377]: 2022-05-14 12:50:06,570 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:50:14 raspberrypi python[377]: 2022-05-14 12:50:14,622 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused mai 14 12:50:21 raspberrypi python[377]: 2022-05-14 12:50:21,314 URL for 'daemon_active' raised and error: cannot connect to ('127.0.0.1', 9080): [Errno 111] Connection refused

pe2269 commented 2 years ago

I try to make a new fresh install of mycodo on one of the raspberry pi zero, but the daemon is still not running. ( i have installed Mycodo on two RPI4 and one RPI3B+, without any problem).

kizniche commented 2 years ago

See the daemon section of the diagnosing issues page: https://github.com/kizniche/Mycodo/wiki/Diagnosing-Issues#daemon-backend

pe2269 commented 2 years ago

Ok this what i got from the debug mod, the daemon doesn't show errors no ? (if we dont take into account when i have activated a conditional controller):

2022-05-18 00:27:17,920 - INFO - mycodo.controllers.controller_input_0f1357d2 - Activated in 4826.4 ms 2022-05-18 00:27:17,921 - DEBUG - mycodo.daemon - Input controller with ID 0f1357d2-c9d9-43c7-8e59-e19bca6359c2 activated. 2022-05-18 00:27:30,940 - INFO - mycodo.controllers.controller_input_b549973e - Activated in 8494.3 ms 2022-05-18 00:27:30,959 - DEBUG - mycodo.daemon - Input controller with ID b549973e-563a-4874-b402-4c6f9b7442d5 activated. 2022-05-18 00:27:30,963 - INFO - mycodo.daemon - All activated Input controllers started 2022-05-18 00:27:30,964 - DEBUG - mycodo.daemon - Starting all activated PID controllers 2022-05-18 00:27:30,965 - INFO - mycodo.daemon - All activated PID controllers started 2022-05-18 00:27:30,966 - DEBUG - mycodo.daemon - Starting all activated Function controllers 2022-05-18 00:27:30,968 - INFO - mycodo.daemon - All activated Function controllers started 2022-05-18 00:27:30,984 - DEBUG - mycodo.daemon - Starting Widget Controller 2022-05-18 00:27:33,361 - DEBUG - mycodo.controllers.controller_widget - Initializing Widgets 2022-05-18 00:27:35,253 - DEBUG - mycodo.controllers.controller_widget - Widgets Initialized 2022-05-18 00:27:35,282 - INFO - mycodo.controllers.controller_widget - Activated in 4294.1 ms 2022-05-18 00:27:35,787 - DEBUG - mycodo.daemon - Widget Controller fully started 2022-05-18 00:27:36,291 - INFO - mycodo.daemon - Mycodo daemon started in 57.010 seconds 2022-05-18 00:27:36,305 - INFO - mycodo.daemon - 54.02 MB RAM in use 2022-05-18 00:28:49,790 - DEBUG - mycodo.stats - Sent anonymous usage statistics 2022-05-18 00:29:42,110 - INFO - mycodo.controllers.controller_conditional_4e4403c7 - Activated in 675.3 ms 2022-05-18 00:29:42,112 - DEBUG - mycodo.daemon - Conditional controller with ID 4e4403c7-a9a4-402d-bab0-be3d7d7ffe0f activated. 2022-05-18 00:29:52,496 - INFO - mycodo.controllers.controller_conditional_4e4403c7 - This INFO log entry will appear in the Daemon Log 2022-05-18 00:29:52,520 - ERROR - mycodo.controllers.controller_conditional_4e4403c7 - This ERROR log entry will appear in the Daemon Log 2022-05-18 00:29:53,789 - ERROR - mycodo.controllers.controller_conditional_4e4403c7 - Exception executing Conditional Statement code Traceback (most recent call last): File "/var/mycodo-root/mycodo/controllers/controller_conditional.py", line 178, in check_conditionals self.conditional_run.conditional_code_run() File "/home/pi/Mycodo/mycodo/user_python_code/conditional_4e4403c7-a9a4-402d-bab0-be3d7d7ffe0f.py", line 30, in conditional_code_run measurement = self.condition("b25d7d13-f138-4ff8-b310-2c3f71de6ed0") File "/var/mycodo-root/mycodo/controllers/base_conditional.py", line 47, in condition return self.control.get_condition_measurement(condition_id) File "/var/mycodo-root/mycodo/mycodo_client.py", line 196, in get_condition_measurement return self.proxy().get_condition_measurement(condition_id) File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 476, in call return self.send(self.name, args, kwargs) File "/var/mycodo-root/env/lib/python3.7/site-packages/Pyro5/client.py", line 243, in _pyroInvoke raise data # if you see this in your traceback, you should probably inspect the remote traceback as well TypeError: 'NoneType' object is not iterable

kizniche commented 2 years ago

So it looks like your daemon is running.

pe2269 commented 2 years ago

Yes the daemon is runnning when, after a reboot of the RPI i do all these steps (because automatically at the boot of the rpi, the daemon is still not runnning): 1° sudo service mycodo stop 2° sudo rm -rf /var/lock/mycodo.pid 3° sudo service mycodo start

The daemon is running but zero measurment are made. i also try with "no device input" like free space, internal temperature ... but nothing !

I have attached the logs :/

mycodo.log mycodoupgrade.log mycododependency.log

pe2269 commented 2 years ago

Ok after activated the debug option in the input configuration, i can see that the controller input add the measurements to InfluxDB with correct values.

But in Mycodo live measurement, i see only no data last XX s

pe2269 commented 2 years ago

I try to start the web UI in debug mode, i obtain these errors :

709-bd35-44fa-866b-26dd68726be3%27+AND+channel%3D%272%27+AND+measure%3D%27dewpoint%27+AND+time+%3E+now%28%29+-+30s&db=mycodo_db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb1d8c870>: Failed to establish a new connection: [Errno 111] Connection refused')) Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 398, in _make_request conn.request(method, url, **httplib_request_kw) File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 239, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/usr/lib/python3.7/http/client.py", line 1260, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1306, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1255, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.7/http/client.py", line 1030, in _send_output self.send(msg) File "/usr/lib/python3.7/http/client.py", line 970, in send self.connect() File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 205, in connect conn = self._new_conn() File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 187, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xb1d8c870>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/adapters.py", line 450, in send timeout=timeout File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 786, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SELECT+LAST%28value%29+FROM+C+WHERE+device_id%3D%27a0883709-bd35-44fa-866b-26dd68726be3%27+AND+channel%3D%272%27+AND+measure%3D%27dewpoint%27+AND+time+%3E+now%28%29+-+30s&db=mycodo_db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb1d8c870>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/mycodo-root/mycodo/mycodo_flask/routes_general.py", line 293, in last_data value='LAST', past_sec=period) File "/var/mycodo-root/mycodo/utils/influx.py", line 182, in query_string raw_data = dbcon.query(query).raw File "/var/mycodo-root/env/lib/python3.7/site-packages/influxdb/client.py", line 527, in query expected_response_code=expected_response_code File "/var/mycodo-root/env/lib/python3.7/site-packages/influxdb/client.py", line 342, in request timeout=self._timeout File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, send_kwargs) File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, kwargs) File "/var/mycodo-root/env/lib/python3.7/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SELECT+LAST%28value%29+FROM+C+WHERE+device_id%3D%27a0883709-bd35-44fa-866b-26dd68726be3%27+AND+channel%3D%272%27+AND+measure%3D%27dewpoint%27+AND+time+%3E+now%28%29+-+30s&db=mycododb (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb1d8c870>: Failed to establish a new connection: [Errno 111] Connection refused')) 2022-05-18 11:20:50,096 192.168.0.132 - - [18/May/2022 11:20:50] "GET /last/a0883709-bd35-44fa-866b-26dd68726be3/input/3a6aab38-a6b8-4ad6-8461-6821f032532e/30?=1652865602196 HTTP/1.1" 204 - 2022-05-18 11:20:52,460 URL for 'last_data' raised and error: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SELECT+LAST%28value%29+FROM+GB+WHERE+device_id%3D%27816da490-f1e8-4bfe-bfe7-781998dead47%27+AND+channel%3D%270%27+AND+time+%3E+now%28%29+-+30s&db=mycodo_db (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xb492d510>: Failed to establish a new connection: [Errno 111] Connection refused')) Traceback (most recent call last): File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/var/mycodo-root/env/lib/python3.7/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa)

pe2269 commented 2 years ago

Is it the connection with influxDB which fail ?

pe2269 commented 2 years ago

I try to delete the influxDB but when i launch the command :

sudo influxd backup /mycodo_db_backup

2022/05/27 10:46:19 backing up metastore to /mycodo_db_backup/meta.00 2022/05/27 10:46:49 Download shard 0 failed copy backup to file: err=read tcp 127.0.0.1:46226->127.0.0.1:8088: read: connection reset by peer, n=0. Waiting 2s and retrying (0)...

pe2269 commented 2 years ago

systemctl status influxd

● influxdb.service - InfluxDB is an open-source, distributed, time series database Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled) Active: activating (start) since Fri 2022-05-27 10:54:43 CEST; 1min 22s ago Docs: https://docs.influxdata.com/influxdb/ Cntrl PID: 15632 (influxd-systemd) Tasks: 8 (limit: 877) CGroup: /system.slice/influxdb.service ├─15632 /bin/bash -e /usr/lib/influxdb/scripts/influxd-systemd-start.sh ├─15633 /usr/bin/influxd -config /etc/influxdb/influxdb.conf └─15849 sleep 1

mai 27 10:55:54 raspberrypi influxd-systemd-start.sh[15632]: ts=2022-05-27T08:55:54.081358Z lvl=info msg="Reading file" log_id=0aiBS9ZW000 mai 27 10:55:54 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 55 attempts... mai 27 10:55:56 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 56 attempts... mai 27 10:55:57 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 57 attempts... mai 27 10:55:58 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 58 attempts... mai 27 10:55:59 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 59 attempts... mai 27 10:56:01 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 60 attempts... mai 27 10:56:02 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 61 attempts... mai 27 10:56:03 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 62 attempts... mai 27 10:56:05 raspberrypi influxd-systemd-start.sh[15632]: InfluxDB API unavailable after 63 attempts...

pe2269 commented 2 years ago

I deleted all influxDB packages and databases, i have reinstalled Mycodo and now it's working great again on Raspberry zero !