v0.8.1 - Set battery reserve, operation mode

[emptywee]

• Added get_mode() function.
• Added set_battery_op_reserve() function to set battery operation mode and/or reserve level. Likely won't work in the local mode.
• Added basic validation for main class __init__() parameters (a.k.a. user input).

[jasonacox]

This is great @emptywee !!!!

Added set_battery_op_reserve() function to set battery operation mode and/or reserve level. Likely won't work in the local mode.

This still allows us to run this as a standalone script (in cloud mode) to change these settings (e.g. cron jobs or embedded in other scrips). Technically (read: Jason hasn't tried) a user of the library can instantiate two instances, one for local, one for cloud and use that one for control commands.

Will a local mode user get enough of an error to indicate they need to swtich to cloud mode? We could add that as a log error.

[emptywee]

Will a local mode user get enough of an error to indicate they need to swtich to cloud mode? We could add that as a log error.

With some modifications to error handling, it can produce:

Testing set_battery_op_reserve()...
DEBUG:https://192.168.1.1:443 "GET /api/operation HTTP/1.1" 200 117
DEBUG:https://192.168.1.1:443 "POST /api/operation HTTP/1.1" 403 115
ERROR:403 Unauthorized by Powerwall API at https://192.168.1.1/api/operation - Endpoint disabled in this firmware or user lacks permission
Set battery operation mode to backup failed.

[emptywee]

@jasonacox ok see https://github.com/jasonacox/pypowerwall/pull/78/commits/8d100faf4bfef47b386b954faf329c86ee61f8f1

[emptywee]

• Added set_mode() function.
• Added set_reserve() function.
• Handle 401/403 errors from Powerwall separately in local mode.
• Handle 50x errors from Powerwall in local mode.

[emptywee]

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

[jasonacox]

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

[emptywee]

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

I'll test with my pw today, see how it behaves.

[emptywee]

Well, in case of token expiration, they respond with 401:

< HTTP/2 401 
< content-type: application/json
< set-cookie: AuthCookie=; Path=/; Max-Age=0
< set-cookie: UserRecord=; Path=/; Max-Age=0
< x-content-type-options: nosniff
< content-length: 38
< date: Wed, 27 Mar 2024 19:22:39 GMT
< 
* Connection #0 to host 192.168.102.64 left intact
{"code":401,"message":"Token Expired"}

Should we adjust the simulation server to mirror this behavior if we want to test how the library re-authenticates?

[jasonacox]

Should we adjust the simulation server to mirror this behavior if we want to test how the library re-authenticates?

Yes, I'm good with that.

[emptywee]

Yes, I'm good with that.

Would you be able to build a new docker image for the pwsim? It is used in PR checks as image: jasonacox/pwsimulator and I cannot update the image in Docker Hub. Unless we want to run the pwsim from the source code of the PR? For that we'll need to change the GitHub Actions workflow, I think.

P.S. I updated the pwsim stub.py to respond with 401 to simulate an expired token.

[emptywee]

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

[jasonacox]

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

I pushed the update for pwsimulator and it passes now. I've always wanted to investigate how to do the automated docker push, but haven't. I have a build script that does it with little effort and typically would only run this when we cut a release.

docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm/v7 --push -t jasonacox/pwsimulator:${VER} .

[emptywee]

OK, now I am not sure, does PW respond with 403 in local mode when session has expired? I'd respond with 401, but who knows? Do you know for sure @jasonacox ? Because your simulator uses 403 to force the example script to "re-login". If PW uses 403 for real to indicate an expired session, I'll have to adjust error handling in this case.

Ugh, you are right. That's been the problem. It bounced between 401 and 403 with previous Firmware versions which is why we eventually went with the range. I haven't check to see what the current version is doing, but I suggest we keep the range or figure out some other logic.

I'll test with my pw today, see how it behaves.

Or maybe the workflow could build a new image and push it Docker Hub automatically each time or if pwsim code was changed, or something along these lines.

I pushed the update for pwsimulator and it passes now. I've always wanted to investigate how to do the automated docker push, but haven't. I have a build script that does it with little effort and typically would only run this when we cut a release.

docker buildx build --no-cache --platform linux/amd64,linux/arm64,linux/arm/v7 --push -t jasonacox/pwsimulator:${VER} .

https://docs.github.com/en/actions/publishing-packages/publishing-docker-images this?

[emptywee]

Nice. I can make some little change to pwsim to test the new gh workflow :)

[emptywee]

@jasonacox hmm, do you need to enable the new workflow somewhere? I didn't see it kicked off on my push to pwsimulator... or it has to be in the main branch?

[emptywee]

Oh, it ran in my/forked repo =)

Weird.

[emptywee]

@jasonacox would you mind setting up a repository level variable IMAGE_REPOSITORY for your repo to jasonacox? It would be at https://github.com/jasonacox/pypowerwall/settings/variables/actions

That way we would be able to run tests/checks in our own repos.

We could go with ${{ github.repository_owner }} but in that case it'll have to match with our docker hub username 1 to 1. Which might be not the case for somebody else who might've forked it to contribute.

[jasonacox]

Done but not sure if I did it right... tests are still failing.

[emptywee]

Weird, it didn't pick it up:

[emptywee]

Why is it working in my repo, but not yours?

[emptywee]

[emptywee]

@jasonacox Ok, it's working for us with ${{ github.repository_owner }}. Let's leave it for now until we figure it out as a separate chore for the repo. What'd you say?

Just didn't want to deviate too much from the original purpose of the PR

[jasonacox]

Crazy. Thanks. I suspect it has to do with not being on the main branch. Anyway, I'll start testing.

[emptywee]

Crazy. Thanks. I suspect it has to do with not being on the main branch. Anyway, I'll start testing.

I don't think so, it picked up the changes in the workflow, but it wouldn't resolve the reference to the variable somehow. It is not in my main branch either, but it works in my repository. No idea what's wrong with GH here.

[jasonacox]

So far, so good. Some minor things...

During testing I noticed that we need to be on TeslaPy >= 24.0. Also, I noticed that calling get_reserve() it seems like force (bypass cache) is not being used:

>>> import pypowerwall
>>> pypowerwall.set_debug(True)
>>> pw = pypowerwall.Powerwall(host="",email=email)
>>> pw.get_reserve(force=True)
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
21.0

I'll see if I can figure out why.

[emptywee]

I can look into it too

[jasonacox]

I added command line arguments for setting/getting mode and reserve. Super handy to test things:

$ python3 -m pypowerwall set -reserve 20
pyPowerwall [0.8.1] - Set Powerwall Mode and Power Levels

Setting Powerwall Reserve to 20

$ python3 -m pypowerwall get 
pyPowerwall [0.8.1] - Set Powerwall Mode and Power Levels

  Site           Tesla Energy Gateway
  Din            1232100-00-E--TG123456789ABC
  Mode           self_consumption
  Reserve        20.0
  Current        41.53570872880039
  Grid           0
  Home           730
  Battery        730
  Solar          0

Feel free to tweak it. :)

[jasonacox]

I created a test container of the Proxy: jasonacox/pypowerwall:0.8.1t51-beta

Problem is that the new validation code is keeping it from starting. I need to see why... the authpath should be set by the PW_AUTH_PATH variable.

03/31/2024 12:30:24 AM [proxy] [INFO] pyPowerwall [0.8.1] Proxy Server [t51] - HTTP Port 8675
03/31/2024 12:30:24 AM [proxy] [INFO] pyPowerwall Proxy Started
03/31/2024 12:30:24 AM [proxy] [ERROR] Directory '.' is not writable. Check permissions.
03/31/2024 12:30:24 AM [proxy] [ERROR] Fatal Error: Unable to connect. Please fix config and restart.

[jasonacox]

Ok, I commented out the file system check and build a new beta container. Now I get a 403 error crashing the proxy:

03/31/2024 12:43:08 AM [proxy] [INFO] pyPowerwall [0.8.1] Proxy Server [t51] - HTTP Port 8675
03/31/2024 12:43:08 AM [proxy] [INFO] pyPowerwall Proxy Started
03/31/2024 12:43:08 AM [proxy] [INFO] pyPowerwall Proxy Server - Local Mode
03/31/2024 12:43:08 AM [pypowerwall.local.pypowerwall_local] [ERROR] 403 Unauthorized by Powerwall API at https://10.0.1.28/api/site_info/site_name - Endpoint disabled in this firmware or user lacks permission
Traceback (most recent call last):
  File "/app/server.py", line 174, in <module>
    log.info("Connected to Energy Gateway %s (%s)" % (host, pw.site_name().strip()))
AttributeError: 'NoneType' object has no attribute 'strip'

I know we thought that we should only see 401 and not 403. But for some reason I'm getting 403's again.

Confirmed: I was getting a 403 to tell me that I needed to refresh my login token. I updated the code to handle 403 and 401's the same and it was able to connect. I'll commit my changes.

Testing: jasonacox/pypowerwall:0.8.1t51-beta3

[jasonacox]

Latest proxy to test: jasonacox/pypowerwall:0.8.1t52-beta1

@mcbirse you may want to try this one... our old friend "SystemConnectedToGrid" has been ressurected 😁

[emptywee]

I know we thought that we should only see 401 and not 403. But for some reason I'm getting 403's again.

Says a lot about tesla api devs hehe...

[emptywee]

I commented out the file system check

Hmm, so the problem is only when running in a container? Might be something odd with the check when path is mounted from the host. I'll look into it, but it's not a blocker for now. I anticipated something like that to happen :-)

[jasonacox]

I commented out the file system check

Hmm, so the problem is only when running in a container? Might be something odd with the check when path is mounted from the host. I'll look into it, but it's not a blocker for now. I anticipated something like that to happen :-)

Actually, the issue was that the proxy was using the cachefile default (local directory .cachefile) which would not be writable. This has been an issue for a long time! It wouldn't surface because the service would just use the password to log in at restart. However, that creates more load on the Powerwall. I updated proxy.

@emptywee your init check code caught this! Love that! ❤️

I added some additional details to the init check messages to help troubleshoot and identify issues. I kept hunting the authpath one but it was really the cachefile one.

[emptywee]

Ah, glad that you find it helpful and not irritating :) was a bit afraid of the latter!

[emptywee]

@jasonacox How's testing going? Ready to merge or not yet?

[jasonacox]

I haven't had time to troubleshoot why the force option is not working with the new functions. It may not be passing it along to the poll(). Do you have time to check?

>>> import pypowerwall
>>> pypowerwall.set_debug(True)
>>> pw = pypowerwall.Powerwall(host="",email=email)
>>> pw.get_reserve(force=True)
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
21.0

[emptywee]

Oh, absolutely. Thought you figured it out already and I was sitting duck waiting for you to merge it :P

[emptywee]

@jasonacox so, I looked at it and it's coming from the part which I didn't really touch, if you check the original code from 0.7.x, e.g.: https://github.com/jasonacox/pypowerwall/blob/7b1fc457d4124727b555cdf05b4fcab4ec412750/pypowerwall/cloud.py#L202-L248 it doesn't honor any kind of force flags there. I can add it there, though. What would you say?

[jasonacox]

You're amazing! Great find. Yes, the main cloud use up to this point was "read only" to get the metric data which is delayed in the cloud anyway, so it hasn't surfaced as an issue. However, with your great work here, we now have "write" functions that could change state that would not show as changed until the cache TTL expires. That doesn't seem ideal. In fact, I wonder if get_reserve() and get_mode() should always be forced by default?

I am still good with merging this but will wait if you want to try to add the force feature first.

[emptywee]

Sure, adding the force flag should be trivial at this point. We can clear certain cache entries once "write" operation is complete.

[emptywee]

@jasonacox I was thinking something like this https://github.com/jasonacox/pypowerwall/pull/78/commits/c46b479b62fcdd3c419be8151d5b20c2cbcf5110

[jasonacox]

Nice @emptywee ! I'm testing...

[jasonacox]

Ok, this is odd. When you set_reserve(), and then try get_reserve() it will return None from cache. Something is wrong with the cache logic. But, if you force=True, it does work.

>>> pw.set_reserve(20)
DEBUG: -- cloud: Request for /api/operation
DEBUG:Invoking 0 protected resource request hooks.
... truncated ...
>>> pw.get_reserve()
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
>>>
>>> pw.get_reserve(force=True)
DEBUG: -- cloud: Request for /api/operation
DEBUG:Invoking 0 protected resource request hooks.
[... trim ...]
DEBUG: -- cloud: Retrieved SITE_CONFIG data
20.0
>>>
>>> pw.get_reserve()
DEBUG: -- cloud: Request for /api/operation
DEBUG: -- cloud: Returning cached SITE_CONFIG data
20.0
>>>

[emptywee]

Weird, I'll double check the caching condition there. Must be something odd there.

[emptywee]

@jasonacox yeah it was a silly thing. Can you check now?

Sorry it took that long, I was taking a shower :)

[emptywee]

Calling it a night now, but it should work now as intended.

[jasonacox]

It works!! Thanks @emptywee !

I'm testing jasonacox/pypowerwall:0.8.1t52-beta4 - should be ready for merge soon.

[jasonacox]

✅ Tests look great. Thanks @emptywee !!! Great enhancements.

[jasonacox]

Library: https://pypi.org/project/pypowerwall/0.8.1/

Proxy Container: jasonacox/pypowerwall:0.8.1t52

[emptywee]

Woot woot!