search

jasonacox / Powerwall-Dashboard

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems
MIT License
270 stars 57 forks source link

Frequent gaps in InfluxDB and errors in pypowerwall #449

Closed jgleigh closed 3 weeks ago

jgleigh commented 3 months ago

I've noticed that lately I'm seeing a lot more missing data in InfluxDB. I run tesla-history periodically to clean things up. I'm guessing this is related to 23.44.0 firmware, but hard to tell. Dashboard up-to-date and running 4.1.1.

Searching InfluxDB for data gaps (power usage)

2024-03-10 15:31:34 03/10/2024 03:31:34 PM [proxy] [INFO] pyPowerwall [0.7.9] Proxy Server [t42] - HTTP Port 8675 2024-03-10 15:31:34 03/10/2024 03:31:34 PM [proxy] [INFO] pyPowerwall Proxy Started 2024-03-10 15:31:36 03/10/2024 03:31:36 PM [proxy] [INFO] pyPowerwall Proxy Server - Local Mode 2024-03-10 15:31:36 03/10/2024 03:31:36 PM [proxy] [INFO] Connected to Energy Gateway 192.168.1.6 (Home Energy Gateway)

Every gap aligns with these messages in the pypowerwall log.

2024-03-10 20:32:10 03/10/2024 08:32:10 PM [proxy] [ERROR] Socket broken sending API response to client [doGET] ... repeats over and over 2024-03-10 20:36:20 03/10/2024 08:36:20 PM [proxy] [ERROR] Socket broken sending API response to client [doGET] 2024-03-10 20:49:30 ---------------------------------------- 2024-03-10 20:49:30 Exception occurred during processing of request from ('172.18.0.6', 34336) 2024-03-10 20:49:30 ---------------------------------------- 2024-03-10 20:49:30 Exception occurred during processing of request from ('172.18.0.6', 34362) 2024-03-10 20:49:30 Traceback (most recent call last): 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:30 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:30 return self._find_no_duplicates(name) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:30 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:30 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:30 2024-03-10 20:49:30 During handling of the above exception, another exception occurred: 2024-03-10 20:49:30 2024-03-10 20:49:30 Traceback (most recent call last): 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:30 self.finish_request(request, client_address) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:30 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:30 self.handle() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:30 self.handle_one_request() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:30 method() 2024-03-10 20:49:30 File "/app/server.py", line 307, in do_GET 2024-03-10 20:49:30 d = pw.system_status() or {} 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:30 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:30 self._get_session() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:30 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:30 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:30 ---------------------------------------- 2024-03-10 20:49:30 Traceback (most recent call last): 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:30 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:30 return self._find_no_duplicates(name) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:30 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:30 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:30 2024-03-10 20:49:30 During handling of the above exception, another exception occurred: 2024-03-10 20:49:30 2024-03-10 20:49:30 Traceback (most recent call last): 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:30 self.finish_request(request, client_address) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:30 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:30 self.handle() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:30 self.handle_one_request() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:30 method() 2024-03-10 20:49:30 File "/app/server.py", line 189, in do_GET 2024-03-10 20:49:30 message = pw.poll('/api/meters/aggregates') 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:30 self._get_session() 2024-03-10 20:49:30 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:30 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:30 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:30 ---------------------------------------- 2024-03-10 20:49:31 ---------------------------------------- 2024-03-10 20:49:31 Exception occurred during processing of request from ('172.18.0.6', 34364) 2024-03-10 20:49:31 Traceback (most recent call last): 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:31 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:31 return self._find_no_duplicates(name) 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:31 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:31 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:31 2024-03-10 20:49:31 During handling of the above exception, another exception occurred: 2024-03-10 20:49:31 2024-03-10 20:49:31 Traceback (most recent call last): 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:31 self.finish_request(request, client_address) 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:31 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:31 self.handle() 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:31 self.handle_one_request() 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:31 method() 2024-03-10 20:49:31 File "/app/server.py", line 192, in do_GET 2024-03-10 20:49:31 message = pw.poll('/api/system_status/soe') 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:31 self._get_session() 2024-03-10 20:49:31 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:31 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:31 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:31 ---------------------------------------- 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38378) 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:35 return self._find_no_duplicates(name) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:35 2024-03-10 20:49:35 During handling of the above exception, another exception occurred: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:35 self.finish_request(request, client_address) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:35 self.handle() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:35 self.handle_one_request() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:35 method() 2024-03-10 20:49:35 File "/app/server.py", line 307, in do_GET 2024-03-10 20:49:35 d = pw.system_status() or {} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:35 self._get_session() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38372) 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:35 return self._find_no_duplicates(name) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:35 2024-03-10 20:49:35 During handling of the above exception, another exception occurred: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:35 self.finish_request(request, client_address) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:35 self.handle() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:35 self.handle_one_request() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:35 method() 2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 38350) 2024-03-10 20:49:35 File "/app/server.py", line 192, in do_GET 2024-03-10 20:49:35 message = pw.poll('/api/system_status/soe') 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:35 self._get_session() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:35 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:35 return self._find_no_duplicates(name) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:35 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:35 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:35 2024-03-10 20:49:35 During handling of the above exception, another exception occurred: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:35 self.finish_request(request, client_address) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:35 self.handle() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:35 self.handle_one_request() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:35 method() 2024-03-10 20:49:35 File "/app/server.py", line 269, in do_GET 2024-03-10 20:49:35 d = pw.system_status() or {} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:35 self._get_session() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:35 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:35 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:35 Exception occurred during processing of request from ('172.18.0.6', 34338) 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 537, in _make_request 2024-03-10 20:49:35 response = conn.getresponse() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 466, in getresponse 2024-03-10 20:49:35 httplib_response = super().getresponse() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 1375, in getresponse 2024-03-10 20:49:35 response.begin() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 318, in begin 2024-03-10 20:49:35 version, status, reason = self._read_status() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/client.py", line 279, in _read_status 2024-03-10 20:49:35 line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socket.py", line 705, in readinto 2024-03-10 20:49:35 return self._sock.recv_into(b) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/ssl.py", line 1307, in recv_into 2024-03-10 20:49:35 return self.read(nbytes, buffer) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/ssl.py", line 1163, in read 2024-03-10 20:49:35 return self._sslobj.read(len, buffer) 2024-03-10 20:49:35 TimeoutError: The read operation timed out 2024-03-10 20:49:35 2024-03-10 20:49:35 The above exception was the direct cause of the following exception: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send 2024-03-10 20:49:35 resp = conn.urlopen( 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen 2024-03-10 20:49:35 retries = retries.increment( 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment 2024-03-10 20:49:35 raise reraise(type(error), error, _stacktrace) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise 2024-03-10 20:49:35 raise value 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen 2024-03-10 20:49:35 response = self._make_request( 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 539, in _make_request 2024-03-10 20:49:35 self._raise_timeout(err=e, url=url, timeout_value=read_timeout) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout 2024-03-10 20:49:35 raise ReadTimeoutError( 2024-03-10 20:49:35 urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='192.168.1.6', port=443): Read timed out. (read timeout=5) 2024-03-10 20:49:35 2024-03-10 20:49:35 During handling of the above exception, another exception occurred: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 197, in _get_session 2024-03-10 20:49:35 r = self.session.post(url,data = pload, verify=False, timeout=self.timeout) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post 2024-03-10 20:49:35 return self.request("POST", url, data=data, json=json, kwargs) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request 2024-03-10 20:49:35 resp = self.send(prep, send_kwargs) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send 2024-03-10 20:49:35 r = adapter.send(request, **kwargs) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 532, in send 2024-03-10 20:49:35 raise ReadTimeout(e, request=request) 2024-03-10 20:49:35 requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='192.168.1.6', port=443): Read timed out. (read timeout=5) 2024-03-10 20:49:35 2024-03-10 20:49:35 During handling of the above exception, another exception occurred: 2024-03-10 20:49:35 2024-03-10 20:49:35 Traceback (most recent call last): 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:35 self.finish_request(request, client_address) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:35 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:35 self.handle() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:35 self.handle_one_request() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:35 method() 2024-03-10 20:49:35 File "/app/server.py", line 269, in do_GET 2024-03-10 20:49:35 d = pw.system_status() or {} 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:35 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:35 self._get_session() 2024-03-10 20:49:35 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 202, in _get_session 2024-03-10 20:49:35 raise ConnectionError(err) 2024-03-10 20:49:35 pypowerwall.ConnectionError: Unable to connect to Powerwall at https://192.168.1.6 2024-03-10 20:49:35 ---------------------------------------- 2024-03-10 20:49:36 ---------------------------------------- 2024-03-10 20:49:36 Exception occurred during processing of request from ('172.18.0.6', 38394) 2024-03-10 20:49:36 Traceback (most recent call last): 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:36 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:36 return self._find_no_duplicates(name) 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:36 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:36 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:36 2024-03-10 20:49:36 During handling of the above exception, another exception occurred: 2024-03-10 20:49:36 2024-03-10 20:49:36 Traceback (most recent call last): 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:36 self.finish_request(request, client_address) 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:36 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:36 self.handle() 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:36 self.handle_one_request() 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:36 method() 2024-03-10 20:49:36 File "/app/server.py", line 189, in do_GET 2024-03-10 20:49:36 message = pw.poll('/api/meters/aggregates') 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:36 self._get_session() 2024-03-10 20:49:36 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:36 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:36 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:36 ---------------------------------------- 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38440) 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:40 return self._find_no_duplicates(name) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:40 2024-03-10 20:49:40 During handling of the above exception, another exception occurred: 2024-03-10 20:49:40 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:40 self.finish_request(request, client_address) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:40 self.handle() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:40 self.handle_one_request() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:40 method() 2024-03-10 20:49:40 File "/app/server.py", line 192, in do_GET 2024-03-10 20:49:40 message = pw.poll('/api/system_status/soe') 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:40 self._get_session() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38422) 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:40 return self._find_no_duplicates(name) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:40 2024-03-10 20:49:40 During handling of the above exception, another exception occurred: 2024-03-10 20:49:40 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:40 self.finish_request(request, client_address) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:40 self.handle() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:40 self.handle_one_request() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:40 method() 2024-03-10 20:49:40 File "/app/server.py", line 307, in do_GET 2024-03-10 20:49:40 d = pw.system_status() or {} 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:40 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:40 self._get_session() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:40 Exception occurred during processing of request from ('172.18.0.6', 38400) 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 210, in _get_session 2024-03-10 20:49:40 self.auth = {'AuthCookie': r.cookies['AuthCookie'], 'UserRecord': r.cookies['UserRecord']} 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 334, in getitem 2024-03-10 20:49:40 return self._find_no_duplicates(name) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/requests/cookies.py", line 413, in _find_no_duplicates 2024-03-10 20:49:40 raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}") 2024-03-10 20:49:40 KeyError: "name='AuthCookie', domain=None, path=None" 2024-03-10 20:49:40 2024-03-10 20:49:40 During handling of the above exception, another exception occurred: 2024-03-10 20:49:40 2024-03-10 20:49:40 Traceback (most recent call last): 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread 2024-03-10 20:49:40 self.finish_request(request, client_address) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request 2024-03-10 20:49:40 self.RequestHandlerClass(request, client_address, self) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/socketserver.py", line 747, in init 2024-03-10 20:49:40 self.handle() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 433, in handle 2024-03-10 20:49:40 self.handle_one_request() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request 2024-03-10 20:49:40 method() 2024-03-10 20:49:40 File "/app/server.py", line 269, in do_GET 2024-03-10 20:49:40 d = pw.system_status() or {} 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 735, in system_status 2024-03-10 20:49:40 payload = self.poll('/api/system_status', jsonformat=True) 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 309, in poll 2024-03-10 20:49:40 self._get_session() 2024-03-10 20:49:40 File "/usr/local/lib/python3.10/site-packages/pypowerwall/init.py", line 219, in _get_session 2024-03-10 20:49:40 raise LoginError("Invalid Powerwall Login") 2024-03-10 20:49:40 pypowerwall.LoginError: Invalid Powerwall Login 2024-03-10 20:49:40 ---------------------------------------- 2024-03-10 20:49:44 03/10/2024 08:49:44 PM [proxy] [ERROR] Socket broken sending API response to client [doGET]

And then everything returns to normal until the next data gap.

jasonacox commented 1 month ago

@jgleigh hold on to your butts! But just to clarify, the epic failure was with the WiFi too or just when you switch back to ethernet cable? The fact that it takes down your entire array feel more than just a cable issue, but I think you are on to something here. If WiFi stays solid, that starts to point more to the Ethernet controller or possibly cable (maybe shorting out and taking controller offline?).

PS - Please watch out for the velociraptor while you get the power back on.

jgleigh commented 1 month ago

Turned on the WiFi and two hours later the Gateway locked up and took down the whole array. Had this happen during the first few months with my system before I routed the hardline. Currently I have both Ethernet and WiFi enabled and I'm pulling the proxy data from the WiFi port. We'll see what happens.

Zero errors in the log too and it's been 7 hours...so hopeful this may be a solution.

jasonacox commented 1 month ago

Anything glitchy like that has me concerned. I do hope it works, however. 🤞

jgleigh commented 1 month ago 
2024-05-16 07:33:00 05/16/2024 07:33:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
2024-05-16 07:57:50 05/16/2024 07:57:50 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
2024-05-16 16:47:25 05/16/2024 04:47:25 PM [pypowerwall.local.pypowerwall_local] [ERROR] 503 Service Unavailable at https://192.168.1.7/api/solar_powerwall - Activating 5 minute API cooldown
2024-05-17 02:27:25 05/17/2024 02:27:25 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
2024-05-17 02:27:30 05/17/2024 02:27:30 AM [pypowerwall] [ERROR] Failed to get /api/system_status/grid_status

Still a few errors, but a VAST improvement. Those were the only times (with repeated messages for 30-60 seconds) since I restarted the proxy on 5/15 at 3:30pm.

And the Ethernet is still connected in case it's causing issues. Think I'm going to disconnect the Ethernet and see what happens for a bit.

jgleigh commented 1 month ago

Much improved. Pulling the Ethernet didn't seem to have any effect good or bad. Still seeing quite a few of these, but much better than before. Anything to look at to see why it's still throwing errors?

2024-05-19 06:30:00 05/19/2024 06:30:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
jgleigh commented 1 month ago

Just had a thought...I have the Plume Mesh WiFi. It constantly optimizes itself to manage all the mesh nodes. I'm wondering if that's injecting errors into the packets as the Gateway has to change channels/bands. The Gateway has shown it doesn't have the most robust networking logic. I'm planning out a switch of the whole system to UniFi so it'll be interesting to see how the stability looks once that's all up and running.

But no reboots since 5/15, so I think I'm on the right track.

jasonacox commented 1 month ago

2024-05-19 06:30:00 05/19/2024 06:30:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe

This is the result of a client (most likely telegraf but could be a browser) timing out and disconnecting from pypowerwall. I say telegraf because it has a 4 second timeout so if the Powerwall does not resupond within 4s, it will disconnect and you will get this error. Technically, we can increase this timeout in the telegraf.conf file to 5s as that is the default for pypowerwall but can also cause some overlap (telegraf samples every 5s). These errors generally aren't an issue as it just means one data point isn't fetched during that 5s sample, but if you see a lot of them, that indicates the Powerwall gateway is having trouble responding within 4s.

Just had a thought...I have the Plume Mesh WiFi. It constantly optimizes itself to manage all the mesh nodes. I'm wondering if that's injecting errors into the packets as the Gateway has to change channels/bands.

Interesting theory. It will be good to see if anything changes when you swap it out.

jgleigh commented 1 month ago

A lot of these have to be telegraf because they're in the middle of night and no browser dashboard is loaded.

BuongiornoTexas commented 1 month ago

Not sure if it is the same thing you are seeing, but I'm on wifi only, and I get periodic socket broken errors in pypowerwall and a continuous stream of context deadline exceeded errors in my telgraf log.

Some of it may be api calls that are no longer valid in telegraf.conf and local.conf, but I suspect a broader problem as well. Haven't had time to investigate properly yet.

jgleigh commented 1 month ago

@BuongiornoTexas mine got exponentially worse on Ethernet (probably a bad cable or network card), so I turned WiFi back on. Could also be something local to my network that's making the issue worse for me.

Same telegraf errors for me: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

cwagz commented 1 month ago

I have been meaning to reply here for several weeks now. I started seeing very large data gaps appearing over the last several weeks. Sometime the gaps would be for multiple hours. My HAProxy would show that the TEG was not responding. The issue would resolve itself and I was using the history script to fill in. This Sunday I noticed the batteries had not charged and that I was pulling from the grid during peak due to the batteries not being used. I logged in to the old TEG interface and there was no error. I used the reset button on the gateway and the batteries have been working again since. I have turned my Powerwall dashboard off to see if the issue returns. Maybe my gateway is going bad or maybe this firmware is very sensitive to the polling from HAProxy and pypowerwall causing lock ups? Or maybe the latest pfSense version has something going on with HAProxy causing the TEG to have issues? There are a lot of variables recently. I have been using HAproxy with the TEG for over a year now without issue. Been on Ethernet the whole time as well.

jasonacox commented 1 month ago

Hi @cwagz - Yikes! What is your Firmware version and what version of Dashboard are you using?

It is hard for me to believe that local API usage would cause problems with the Gateway and batteries. It is all read-only and It typically will respond with rate limiting errors if you hit it too hard. I have Firmware 24.4.0 and 3 different Dashboards (all current Firmware) polling my gateway without any issue. I'm also using current base install (not HAProxy).

While you wait to see if it returns, you can also switch your Dashboard to cloud mode so at least you have current state if you want access to it. The downsize to cloud mode is that they have removed the Powerwall capacity data points (sadly also gone from FleetAPI) which means the only way to get Powerwall degradation data is via local API (/api/system_status).

cwagz commented 1 month ago

Hi @jasonacox - I was on the May 15th version of the dashboard and firmware 24.4.0. I have a bad feeling that something must be going wrong with my gateway. I was going to give it a couple weeks to see if any weirdness happens again. I trust what you are saying regarding the negligible effect the dashboard should have.

jgleigh commented 1 month ago

I had 10 days with no issues and then 2 reboots in the last 12 hours. The randomness is just crazy!

@cwagz I've also got a thread going on TMC: https://teslamotorsclub.com/tmc/threads/24-4-0-gateway-crashes.326493/

jgleigh commented 1 month ago

Back to Ethernet I guess...what a crappy network module these things have!!!

Screenshot 2024-05-28 at 10 00 14 AM
jasonacox commented 1 month ago

Ugh. Sorry to hear that @jgleigh

My system has weak WiFi even with a mesh node close enough to give it a strong signal. I would see 5m spans where all packets would drop, but nothing like you are seeing. All of that went away when I hardwired in ethernet. Your gateway has me puzzled. I've not been able to throw enough traffic, or bad traffic, at my Gateway to get it to reboot. It will throttle me with 429 HTTP rate limit errors, but it never goes offline.

When I run ethernet cable, I always use a testing tool to ensure I don't short (or long) any of the wires during crimping. I'm not an expert crimper, so it is required. 😂

🤞 that this time goes better for you.

cwagz commented 1 month ago

@jgleigh, I have HAProxy still monitoring my TEG and it is still dropping out on Ethernet as well every few days. I have not had to reset it again though. No effect with production or charging / discharging again. I will keep an eye on your TMC post as well. Does seem coincidental with the 24.4.0 firmware. I will get my dashboard back up and running. Tired of feeling blind.

jasonacox commented 1 month ago

@cwagz how long does it go offline? Also, @jgleigh does it seems that everything reconnects when it comes back online?

If there was only a way to get Vitals data... 🤷

cwagz commented 1 month ago

@jasonacox - I will get a screenshot of it on the dashboard once it happens again. It would just flatline all the data and pick right back up after sometimes hours. The data would strangely load from the cloud though if I ran the history script. Then the one time my powerwalls stopped charging / discharging and I had to use the reset button to get them working again. I have not had this happen again yet and although I am seeing drops in the haproxy stats there is no data missing in the Tesla app. Probably because it is using the cloud data which I agree must buffer inside the gateway somehow.

BuongiornoTexas commented 1 month ago

@jgleigh - A long shot here (definitely mentioned elsewhere but not in this thread yet) - my powerwall disappears from my network if I don't use a static ARP. I've had to set this up on both my dashboard box and my windows box. If I don't do this, my powerwall stops responding to TCP traffic on the local network.

The way I used to get around it before figuring this out was going into the app and forcing it to reconnect to the local network.

jasonacox commented 1 month ago

Thanks @cwagz. The local data drop is consistent with my tests where I overload the API (as I mentioned, getting 429 HTTP errors) and it will take 5-10m to "cool down" and let me back in. but, I have never seen the System stop functioning (charge/discharge). Maybe I'm just lucky and there is a bug in 24.4.0 that I just haven't hit yet. It would be extremely noisy, but you could activate DEBUG mode in pypowerwall.env just in case there is some clue as to what is happening. But to be fair, I would think your HAProxy would have just as much info (if not more).

I seriously doubt this applies to anyone else, but I run multiple copies of Powerwall-Dashboad across Linux, RPI, Windows, Mac hosts. I originally started seeing the "data gaps" and 429 errors after the removal of vitals. It was due to the retry logic in pypowerwall trying to fetch vitals. Even most recently, I had a rogue system that was still running the old code and would eventually cause the gateway to rate limit local traffic. As I say, I doubt that scenario would apply for anyone else in the community but putting it out there just in case.

Thanks @BuongiornoTexas for your note as well.

I'm sure they have their reasons, but I'm disappointed in what we are seeing in the firmware updates. The removal of vitals and not providing a local API for the PW3 feel like we are going backwards. If it is related to the system failures you and @jgleigh have raised, that is even more concerning. Maybe the next release will improve things. In the meantime, I'm working on some things... 😁

jgleigh commented 1 month ago

@BuongiornoTexas Both Ethernet and WiFi are assigned fixed addresses so they don't move around. When the WiFi locked up yesterday, the Ethernet was still up and functional. I did reboot the system last night so both interfaces are up and running.

@jasonacox The total system shutdown is fairly rare. Was much more common when I first got the system installed three years ago. Once switching to Ethernet it was pretty rock solid until around the time they started removing stuff from the API (23.44.x). I'm keeping both interfaces up again to see what happens and polling from the Ethernet one. I've also been messing with my home network a lot so nothing is truly stable at the moment. Still planning on rebuilding my WiFi APs, but I'm not quite there yet (wife is poking me about all the money I'm spending). I should also run a cable test on the Ethernet line since it spans the entire length of my house from the switch to the Gateway, but haven't gotten around to that yet. Got plenty of critters around here that like to chew on things!

Very unfortunate they've locked down the PW3s so much. None of us would have any knowledge about our systems or how to troubleshoot them without getting into the API. Sounds like they may have blocked people from using Tesla One on PW3s as well. Imagine trying to get Tesla tech support on the line and doing something useful instead of just rebooting the whole system. I would have been down for months at a time if I couldn't restart the system locally from the installer interface.

BuongiornoTexas commented 1 month ago

| Both Ethernet and WiFi are assigned fixed addresses

@jgleigh - Just to be clear we are on the same page, I'm talking about ARP - not static IP addresses. It's part of the protocol layer that links IP address to MAC address. This fix shouldn't be required - for almost everything ARP should just work without user intervention (to the point I'd be surprised if you were tweaking your ARP tables at all!) - but in my case I get consistent drop outs after a few hours without it, and no problems at all with it.

(one of the similarities between my set up and your is that both ethernet and wifi are enabled on the powerwall - even though I'm not using ethernet, I suspect the multiple active networks is causing the ARP problem).

cwagz commented 1 month ago

| Both Ethernet and WiFi are assigned fixed addresses

@jgleigh - Just to be clear we are on the same page, I'm talking about ARP - not static IP addresses. It's part of the protocol layer that links IP address to MAC address. This fix shouldn't be required - for almost everything ARP should just work without user intervention (to the point I'd be surprised if you were tweaking your ARP tables at all!) - but in my case I get consistent drop outs after a few hours without it, and no problems at all with it.

(one of the similarities between my set up and your is that both ethernet and wifi are enabled on the powerwall - even though I'm not using ethernet, I suspect the multiple active networks is causing the ARP problem).

I am going to give this a shot. Maybe something weird in the latest pfsense version that made this crop up.

BuongiornoTexas commented 1 month ago

| I am going to give this a shot

I've got some notes on how I set it up somewhere - let me know if you would like me to share them. (for both windows and one of the linux implementations).

jasonacox commented 1 month ago

let me know if you would like me to share

Please do, @BuongiornoTexas ! I suspect you aren't the only one with that issue.

jgleigh commented 1 month ago

@BuongiornoTexas I'm assigning fixed IP addresses to the WiFi and Ethernet modules through DHCP reservations.

BuongiornoTexas commented 1 month ago

I'm assigning fixed IP addresses to the WiFi and Ethernet modules through DHCP reservations.

@jgleigh - DHCP reservations are effectively static IP (I'm also using these).

The ARP issue is something else again. I'll post how to do checks on that tomorrow (sadly, with the way the firmware is now, not any guarantee this will help, but worth eliminating as a possible issue at the very least). Which platform are you on?

@jasonacox - I'll post what I've done in discussions and cross link here - it'll be easier to find for future reference and can be linked in the docs if more people than me find it useful (although I'm getting used to being an edge case ;-)).

jgleigh commented 1 month ago

@BuongiornoTexas Dashboard is running on macOS and using UniFi UDM Pro for my router.

BuongiornoTexas commented 1 month ago

Dashboard is running on macOS

Dashboard machine is the key - I'll see if I can find docs on the macos approach as well then.

cwagz commented 1 month ago

I already had static arp assigned in pfsense for everything on my network. Both Ethernet and WiFi are connected from the TEG and they both are in the arp table as permanent. I will have the dashboard back up soon but am still unsure what is going on with my TEG networking. I could move the WIFI to an isolated vlan with internet access only and see if that helps. Maybe the teg does not do well having both connected to the same network with the newer firmware.

BuongiornoTexas commented 1 month ago

I already had static arp assigned in pfsense for everything on my network.

@cwagz Well, I hope that eliminates it as a possible cause for the problem for you. But I do note that my ARP issue happens in spite of using a static DHCP allocation with MAC address on the router - the ARP problem happens on the client machines trying to access the powerwall, and this seems to be independent of what the router knows (if my weak understanding is correct, each machine maintains its own ARP table?).

Still digging out the docs and unfortunately distracted by other IRL things. Hopefully get them up in a day or two.

BuongiornoTexas commented 1 month ago

I've put together a discussion thread outlining the static arp thing at #480.

@jgleigh - Unfortunately, I haven't had time to do more than a cursory check on setup for macos - joys of other things going on. I will hopefully update this in a couple of days or less.

BuongiornoTexas commented 1 month ago

@jgleigh After doing a bit more digging, I'm pretty sure the macos solution will be using some variant on the arp command. However, I have no idea how to get it set up permanently.

So before digging into that, it's probably worth doing some trials to see if your problem is arp related. The first thing I'd do is try the following from a machine that is losing access to the powerwall:

arp -a <powerwall ip address>

If your machine can currently reach the powerwall, it should show something like "reachable" or "dynamic" in the status line. If the powerwall is not responding and it is the arp issue, then you will see a "stale" status. If the latter is happening, then we may the have the arp problem and can try to figure a solution from there. (Otherwise, it's probably one of the other manifold issues that seem to afflict the powerwall wifi networking, and all we can do is keep looking ...).