search

jasonacox / Powerwall-Dashboard

Grafana Monitoring Dashboard for Tesla Solar and Powerwall Systems
MIT License
268 stars 57 forks source link

Gateway password resets (v4.1.3) #456

Open jasonacox opened 3 months ago

jasonacox commented 3 months ago

I'm having to reset my gateway password everyday or so. Says Bad Credentials. I have the latest version 4.1.3; did this version have a fix for this? Could the cause be it polling solar_powerwall when I don't have Powerwall+? Cheers

Originally posted by @simonrb2000 in https://github.com/jasonacox/Powerwall-Dashboard/discussions/109#discussioncomment-8894034

jasonacox commented 3 months ago

Hi @simonrb2000 - Creating an issue for this.

Where are you resetting the password? On the Powerwall gateway itself?

Can you give us a sample of the log file you see for pypowerwall?

docker logs pypowerwall
simonrb2000 commented 3 months ago

Thanks Jason,

On the gateway IP. Prior to 4.1.3 it would need updating every couple of hours.

It only needs reseting to log back into gateway as the Powerwall-Dashboard remains logged in. It's only when I restart Powerwall-Dashboard does it become a problem. Hope that makes sense?

Basically, I can't log into the gateway as it comes up with bad credentials and I have to reset the password then redo the setup for Powerwall-Dashboard.

If I'm not fussed about logging into the gateway then I can leave it for days and days and its only a problem when I restart Powerwall-Dashboard.

Thanks again for opening an issue and apologise for not doing it myself

jasonacox commented 3 months ago

Thanks for raising this issue @simonrb2000 - others may have the same problem.

When you "reset" the password, are you using an old password to log in? If so, i wonder, does it work if you use the old password in Powerwall-Dashboard?

simonrb2000 commented 3 months ago

No, I've been using a new one each time. I have to select forgotten password as an option and the last 5 characters from Gateway serial and then toggle the Powerwall switch. Only then do I need to change the password on Powerwall-dashboard

jasonacox commented 3 months ago

Thanks. I'm sorry for so many question. I would like to understand the exact steps you are taking so we can see if there a way to make it "stick". It feels like this is a problem with the Gateway "forgetting".

I'm guessing this is what you are doing - the https://10.0.1.99 being the IP address of the Powerwall:

  1. Go to Gateway portal https://10.0.1.99/
image
  1. Click Forgot Password - then check "Forgot password"
image

After you enter a new password (and do the toggle or other instruction?)

  1. Now, after resetting the password, if you go back to https://10.0.1.99 , are you able to log in using that new password?
simonrb2000 commented 3 months ago

That's exactly it. It only seems to happen when I'm running Powerwall-Dashboard. If I disable it then I don't have the issue.. can be a week later and the password would still work. I'm only putting 2 and 2 together and maybe not getting 4. V4.1.3 has helped but instead of everyday it's every few days it needs resetting now.

jasonacox commented 3 months ago

Thanks. I'm not able to replicate. Is anyone else seeing this? It might be load from the dashboard, but I have 5 different Powerwall-Dashboard instances running all hitting my gateway all the time, and no issues.

Did your system upgrade to Firmware version 24.4.0?

simonrb2000 commented 3 months ago

Could it be that it’s asking for solar info and it doesn’t have it? Is there a way for me to stop it polling for solar info?

Yes, I reported 24.4.0 to you 😁 although it did happen on the previous version.

jasonacox commented 3 months ago

I reported 24.4.0 to you 😁

I apologize. My context windows is too small and I have too many concurrent threads / projects. 😊

stop it polling for solar info

It polls for these as defined in the file telegraf.conf https://github.com/jasonacox/Powerwall-Dashboard/blob/f6d829f5c55946a0154673990482bb67e8f9706a/telegraf.conf#L101-L109

You could remove the "strings" one as it is the only one that is exclusive for solar. The rest contain payload data for the powerwall, home and grid.

I'm sure you already sent this at some point, but can you paste a sample list of your pypowerwall logs to see anything unusual?

docker logs pypowerwall
jasonacox commented 3 months ago

@simonrb2000 there is a new version of pypowerwall about to be released. It has some optimizations and refactoring. It may be interesting to see if it makes a difference for your system. You can give it a try:

Edit the powerwall.yml file and change the pypowerwall section to update the image:

    pypowerwall:
        image: jasonacox/pypowerwall:0.8.0t51-beta3
        container_name: pypowerwall
        hostname: pypowerwall
        restart: unless-stopped

Restart the stack:

/compose-dash.sh up -d
KevM commented 3 months ago

I am seeing this problem too for a few weeks. I didn't notice this issue until now. I updated to beta3 of pypowerwall and will report in if the reset problem goes away.

jasonacox commented 3 months ago

In out preparation for the new pypowerwall release, we did find an issue with how the proxy caches the login session causing it to attempt to log in when pypowerwall restarts. I don't think this is your particular use case, but you could try the updated beta if you are interested: jasonacox/pypowerwall:0.8.1t52-beta1

simonrb2000 commented 3 months ago

Sorry, I’ve been away and not had a chance to look at this, I’ll be back soon and I’ll update how it goes. Ta

On 1 Apr 2024, at 05:07, Jason Cox @.***> wrote:



In out preparation for the new pypowerwall release, we did find an issue with how the proxy caches the login session causing it to attempt to log in when pypowerwall restarts. I don't think this is your particular use case, but you could try the updated beta if you are interested: jasonacox/pypowerwall:0.8.1t52-beta1

— Reply to this email directly, view it on GitHubhttps://github.com/jasonacox/Powerwall-Dashboard/issues/456#issuecomment-2029122495, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2AEAGLTK326CUHYBVL7ITY3DMQLAVCNFSM6AAAAABFF3FXZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGEZDENBZGU. You are receiving this because you were mentioned.Message ID: @.***>

KevM commented 3 months ago

Updated to jasonacox/pypowerwall:0.8.1t52-beta1 today. When I restarted I noticed that the gateway had locked me out again (likely pre update). At the gateway web page I am receiving this error: API Limit Reached. So maybe for my situation pypowerwall is hammering the gateway too hard? Is there a way to access or turn on logging so I can retrieve extra logs for your use?

simonrb2000 commented 3 months ago

That’s the same error I receive mostly. Occasionally it says bad credentials even though they are correct.

On 1 Apr 2024, at 16:01, Kevin Miller @.***> wrote:



Updated to jasonacox/pypowerwall:0.8.1t52-beta1 today. When I restarted I noticed that the gateway had locked me out again (likely pre update). At the gateway web page I am receiving this error: API Limit Reached. So maybe for my situation pypowerwall is hammering the gateway too hard? Is there a way to access or turn on logging so I can retrieve extra logs for your use?

— Reply to this email directly, view it on GitHubhttps://github.com/jasonacox/Powerwall-Dashboard/issues/456#issuecomment-2029900232, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2AEAFU4JEYKFLKS2M6GM3Y3FZEVAVCNFSM6AAAAABFF3FXZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZHEYDAMRTGI. You are receiving this because you were mentioned.Message ID: @.***>

KevM commented 3 months ago

Logging output to a file manually via docker logs -f pypowerwall > pypowerwall-0401.log If the password reset problem occurs I'll relay the logs.

jasonacox commented 3 months ago

By chance are you running any other application or scripts that access the Powerwall?

KevM commented 3 months ago

I do not. Only Powerwall-Dashboard. My logs from yesterday were filled with this: 

04/02/2024 05:15:55 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
04/02/2024 05:16:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
04/02/2024 05:16:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
04/02/2024 05:16:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
04/02/2024 05:16:00 AM [proxy] [ERROR] Socket broken sending API response to client [doGET]: [Errno 32] Broken pipe
----------------------------------------
Exception occurred during processing of request from ('172.21.0.6', 49902)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()
  File "/usr/local/lib/python3.10/http/server.py", line 433, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.10/http/server.py", line 421, in handle_one_request
    method()
  File "/app/server.py", line 318, in do_GET
    fcv["grid_status"] = pw.grid_status(type="numeric")
  File "/app/pypowerwall/__init__.py", line 616, in grid_status
    grid_status = payload['grid_status']
TypeError: 'NoneType' object is not subscriptable
----------------------------------------
jasonacox commented 3 months ago

Unfortunately, the error is expected if the Powerwall stops responding.

This is very puzzling. The basic pypowerwall setup is designed to cache data to prevent heavy loads on the Powerwall. I can't replicate this even after having 5 instances of pypowerwall running. It would be good to get more debug information to see if there is any other indication on what is happening. If you want to do that:

Edit pypowerwall.env and turn PW_DEBUG=yes:

PW_EMAIL=email@example.com
PW_PASSWORD=password
PW_HOST=10.0.1.2
PW_TIMEZONE=America/Los_Angeles
PW_DEBUG=yes
PW_STYLE=grafana-dark

Restart and watch logs...

/compose-dash.sh up -d; docker logs pypowerwall -f
simonrb2000 commented 3 months ago

Could this be a powerwall vice powerwall + thing? I have 2 x Powerwall with Gateway 2.

What do you guys have?

On 3 Apr 2024, at 03:55, Jason Cox @.***> wrote:



Unfortunately, the error is expected if the Powerwall stops responding.

This is very puzzling. The basic pypowerwall setup is designed to cache data to prevent heavy loads on the Powerwall. I can't replicate this even after having 5 instances of pypowerwall running. It would be good to get more debug information to see if there is any other indication on what is happening. If you want to do that:

Edit pypowerwall.env and turn PW_DEBUG=yes:

@.*** PW_PASSWORD=password PW_HOST=10.0.1.2 PW_TIMEZONE=America/Los_Angeles PW_DEBUG=yes PW_STYLE=grafana-dark

Restart and watch logs...

/compose-dash.sh up -d; docker logs pypowerwall -f

— Reply to this email directly, view it on GitHubhttps://github.com/jasonacox/Powerwall-Dashboard/issues/456#issuecomment-2033439121, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD2AEAEK2DVR3IWPSURS643Y3NVRHAVCNFSM6AAAAABFF3FXZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZTGQZTSMJSGE. You are receiving this because you were mentioned.Message ID: @.***>

KevM commented 3 months ago

I have a single Powerwall 2 and gateway. I restarted today (it had stopped working again) with debug on and am watching the logs. I thought it was interesting that there is a connection pool of size 3 connections (if you are avoiding stressing the gateway why hit it 3 ways at a time?) and they seem to churn a lot. Here are the logs from the startup of Pypowerwall and first 30 seconds: 

2024-04-03 07:28:09 04/03/2024 07:28:09 AM [proxy] [INFO] pyPowerwall Proxy Started
2024-04-03 07:28:09 04/03/2024 07:28:09 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Tesla local mode enabled
2024-04-03 07:28:09 04/03/2024 07:28:09 AM [pypowerwall.local.pypowerwall_local] [DEBUG] loaded auth from cache file .auth/.powerwall (cookie authmode)
2024-04-03 07:28:09 04/03/2024 07:28:09 AM [proxy] [INFO] pyPowerwall Proxy Server - Local Mode
2024-04-03 07:28:09 04/03/2024 07:28:09 AM [proxy] [INFO] Connected to Energy Gateway 192.168.7.87 (Harrison)
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.7.87/api/devices/vitals
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.7.87/api/devices/vitals
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.7.87/api/devices/vitals
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 234400 detected - Does not support vitals API - disabling.
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:15 04/03/2024 07:28:15 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:16 04/03/2024 07:28:16 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:20 04/03/2024 07:28:20 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:21 04/03/2024 07:28:21 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:21 04/03/2024 07:28:21 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:25 04/03/2024 07:28:25 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:26 04/03/2024 07:28:26 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:30 04/03/2024 07:28:30 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:31 04/03/2024 07:28:31 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:35 04/03/2024 07:28:35 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:40 04/03/2024 07:28:40 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /alerts/pw HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /soe HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /freq HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /temps/pw HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /pod HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [proxy] [DEBUG] 172.21.0.6 "GET /aggregates HTTP/1.1" 200 -
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall
2024-04-03 07:28:45 04/03/2024 07:28:45 AM [urllib3.connectionpool] [WARNING] Connection pool is full, discarding connection: 192.168.7.87. Connection pool size: 3
2024-04-03 07:28:46 04/03/2024 07:28:46 AM [proxy] [DEBUG] 172.21.0.6 "GET /strings HTTP/1.1" 200 -
2024-04-03 07:28:46 04/03/2024 07:28:46 AM [pypowerwall.local.pypowerwall_local] [DEBUG] Empty response from Powerwall at https://192.168.7.87/api/solar_powerwall

The empty responses seem worrisome. Also hitting every route on the gateway every 5 seconds seems aggressive. Is there a setting to dial this back?

KevM commented 3 months ago

Powerwall had stopped working for my login. I reset the password via the gateway. What is interesting is I noticed that the contents of the cached AuthCookie in the .auth file did not change and pypowerwall was able to get connected.

04/03/2024 07:28:09 AM [pypowerwall.local.pypowerwall_local] [DEBUG] loaded auth from cache file .auth/.powerwall (cookie authmode)

So it appears that even though the account password was cycled the auth cookie didn't need to be refreshed? Likely not related to the problem but very curious.

jasonacox commented 3 months ago

Thanks @KevM - I have a Powerwall + and Powerwall 2. I agree about the concern about the empty response.

Is there a setting to dial this back?

Yes, there is! You can set this in pypowerwall.env by adding:

# Disable connection pooling - default is 15
PW_POOL_MAXSIZE=0

# You could increase the cache duration - default is 5s
PW_CACHE_EXPIRE=10

Restart and watch logs...

/compose-dash.sh up -d; docker logs pypowerwall -f

So it appears that even though the account password was cycled the auth cookie didn't need to be refreshed? Likely not related to the problem but very curious.

Yes, that is true. It continues to honor the auth until the TTL is reached. Now, if you changed the password in the gateway portal but didn't update the password in pypowerwall.env (or via setup.sh) and restart pypowerwall, it would still connect until the TTL was reached and then would be unable to re-auth, causing it to loop retry which would eventually hit the API limit. I recommend deleting the .auth files and restart pypowerwall to see if is able to connect.

jasonacox commented 3 months ago

Also, you may switch to this new beta version that adds additional DEBUG logging showing details on cache vs polling: jasonacox/pypowerwall:0.8.1t52-beta3

KevM commented 3 months ago

Running new beta3 with the following .env 

// ...  credentials ...
PW_TIMEZONE=America/Chicago
PW_STYLE=grafana-dark
TZ=America/Chicago
PW_DEBUG=yes
PW_CACHE_EXPIRE=30
PW_POOL_MAXSIZE=1

I have two timezones in there for some reason. Maybe that is from the tz.sh script.

Will let you know if this fails and send logs.

KevM commented 3 months ago

One day later and we are still good (no forced password reset). I think the lower connections, or the longer expiration may be helping.

jasonacox commented 3 months ago

That's great news @KevM . I know you have debug on so it is probably filling up your disk fast. But it would be good for you to sample the logs to see if you see anything unusual.

KevM commented 2 months ago

It has been running well now for 4 days. Longest I can remember. Here is a gist of a log sample.

jasonacox commented 2 months ago

Thanks @KevM - I'm glad you found a fix! Part of me wants to know if it was PW_CACHE_EXPIRE or PW_POOL_MAXSIZE that fixed it, or both? Thanks for the update. Hopefully it will help others seeing the same thing.

KevM commented 2 months ago

My guess it maybe a combination. Could be an API limit and the 5 second interval creeps outside of it. Or maybe the concurrent access triggers an internal limit. Of course, I’m not sure. I’m happy to try just one of the settings. Any suggestions?

On Mon, Apr 8, 2024, at 10:02 PM, Jason Cox wrote:

Thanks @KevM https://github.com/KevM - I'm glad you found a fix! Part of me wants to know if it was PW_CACHE_EXPIRE or PW_POOL_MAXSIZE that fixed it, or both? Thanks for the update. Hopefully it will help others seeing the same thing.

— Reply to this email directly, view it on GitHub https://github.com/jasonacox/Powerwall-Dashboard/issues/456#issuecomment-2044068861, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAMHL63YLHGCPHLL73VSDY4NK5XAVCNFSM6AAAAABFF3FXZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUGA3DQOBWGE. You are receiving this because you were mentioned.Message ID: @.***>

billraff commented 2 months ago

Not sure if I saw this same issue but I just this morning noticed that on my two instances of Powerwall-Dashboard neither had been collecting data for the last 3 days. The verify script show pypowerwall was down and the logs indicated an issue with authentication. Several days ago I had received notification in my Home Assistant Tesla integration that I needed to re-authenticate. I did so with no problem but never thought to check other applications to see if they needed me to re-authenticate. I do use the free iOS app Netzero on my iPad (highly recommend) and check it daily and it never has asked for me to authenticate.

KevM commented 2 months ago

@billraff Did you need to reset your gateway password after the authentication issue? That was what was happening to me. My resolution so far was so slow pypowerwall down (30s) and use less (1) connections in the pool.

billraff commented 2 months ago

I did go to the Gateway and it asked me to login. It did ask me to reset the password but honestly it seems I almost always have to do that there. It's pretty infrequent I log in to it. I'll keep an eye on things for a few days.

jasonacox commented 2 months ago

Hi @billraff - thanks for the updates. What version of pypowerwall are you running (shows in verify.sh)? Several recent updates have been made to handle the newer Powerwall firmware that removed some API endpoints.

I do use the free iOS app Netzero on my iPad (highly recommend) and check it daily and it never has asked for me to authenticate.

Yes, Netzero, like the Tesla App, uses the Tesla cloud endpoint to get data so it would not be impacted by the local authentication issues. You can switch pypowerwall into cloud mode but you will lose some of the data only available in local mode and the fidelity is less (the cloud caches data for longer periods of time). The Home Assistant integration is probably using local access similar to pypowerwall.

billraff commented 2 months ago

I am running 4.2.1 on both my systems. It happened again yesterday where it appears I was locked out of the gateway. I noticed this because in Home Assistant there was a notification from the Powerwall integration that it needed fixing. It wanted me to provide credentials to the gateway. Before doing so I looked at my two Powerwall-Dashboards and saw neither had been collecting data for about an hour. I fixed the integration issue by providing gateway credentials and then using a browser went to the gateway and logged in there. It wanted me to reset the password which I did and both dashboards began working again. I've disabled the Tesla Powerwall integration (https://www.home-assistant.io/integrations/powerwall) and the Tesla Custom Integration that is installed through HACS. I'll watch for a week or so and then if all is well enable the Powerwall one to observe behavior. If all is well for another week then the custom integration.