search

jasonacox / pypowerwall

Python API for Tesla Powerwall and Solar Power Data
MIT License
134 stars 24 forks source link

v0.9.0 - FleetAPI #91

Closed jasonacox closed 4 months ago

jasonacox commented 4 months ago

pyPowerwall v0.9.0 - Tesla FleetAPI Support

This PR adds the official Tesla FleetAPI as an option to connect with the Tesla Cloud. The unofficial Tesla Owners API has been discontinued for newer Tesla vehicles and will likely be shutdown soon for others, including Powerwall and Solar owners. This is an attempt to get ahead of that shutdown and provide an equivalent cloud mode option for pypowerwall users.

Setup

Getting set up to use the FleetAPI from Tesla requires registration and a several step process (TODO: Update readme). See setup steps here for now: https://github.com/jasonacox/pypowerwall/tree/main/tools/fleetapi#tesla-developer---fleetapi-for-powerwall

You will run this to get the configuration set up for pypowerwall:

# Run FleetAPI configuration
python3 -m pypowerwall fleetapi

Testing

Library testing in python:

# Use FleetAPI to connect
import pypowerwall
pw = pypowerwall.Powerwall(fleetapi=True)
pw.power()

Command line testing:

# See CLI options
python3 -m pypowerwall.fleetapi

# See status
python3 -m pypowerwall.fleetapi status

TODO List

Related Issues and Discussions

https://github.com/jasonacox/pypowerwall/discussions/71 https://github.com/jasonacox/Powerwall-Dashboard/issues/425

Also big thanks to @emptywee for pre-work on restructuring the code so this addition could be much easier.

emptywee commented 4 months ago

This is great, @jasonacox ! Love seeing my changes leading to a more developed, flexible and mature library! Eventually, we should provide an auto mode for users so that the library will automatically pick the best class suitable for the user request, based on which modes (local, cloud, fleetapi, etc) are configured. And maybe even fallback to other mode if the primary/best isn't available for some reason.

jasonacox commented 4 months ago

Love that idea @emptywee ! I added an auto-select init argument that will check to see which is available in this order:

1) Local - Requires:host and email 2) FleetAPI - Requires setup (python -m pypowerwall fleetapi) 3) Cloud - Requires setup (python -m pypowerwall setup)

Currently automatic selection is only done once during instantiation.

import pypowerwall

pw = pypowerwall.Powerwall(auto_select=True)
pw.power()
jasonacox commented 4 months ago

Proxy updated to support *Auto Select mode. I was able to verify that proxy can use FleetAPI. Beta container:

jasonacox/pypowerwall:0.9.0t56-beta2

jasonacox commented 4 months ago

Testing: jasonacox/pypowerwall:0.9.0t56-beta9

Setup FleetAPI via docker:

docker exec -it pypowerwall python3 -m pypowerwall fleetapi
jasonacox commented 4 months ago

@mcbirse If you have time, try this out and let me know if it works in your setup.

mcbirse commented 4 months ago

@jasonacox - Will do, hopefully I can find some time to try it out in the next few days. I haven't been able to contribute much lately unfortunately due to other commitments. Quite frustrating to be honest! 😞

Interestingly, just struck an issue that probably needs addressing with the pypowerwall repository though (only affects Windows users... but that includes me, so my preference is to fix this 😄 ).

I tried to checkout/clone pypowerwall again and I am no longer able to! I was using VSCode in Windows and the checkout fails. Tried with Git Bash out of curiosity and also fails. The problem is Windows.

error: invalid path 'pypowerwall/aux.py'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.

Apparently Windows cannot handle files named "aux.*" as AUX is reserved (along with many more).

Recommended / easy fix is to just rename the file in the repository (and ensure not to use any reserved filenames). Further details on this issue are here.

Up to you. I can work around it of course since I am using a Linux host for pypowerwall / Powerwall-Dashboard.

However, I do still use VSCode in Windows for coding / review etc. And, I was thinking this may actually affect some users on the Git Bash / Docker Desktop environment if they were trying to checkout and test pypowerwall directly.

spoonwzd commented 4 months ago

Yeah I came up against the AUX issue in Windows too - was news to me and I've been in the industry using Windows for over 30 years! Unraid docker to the rescue.

jasonacox commented 4 months ago

Recommended / easy fix is to just rename the file in the repository (and ensure not to use any reserved filenames). Further details on this issue are here.

Agree - let's change the name. Wild!!! Good discovery.

And.... I found an issue with FleetAPI token renew logic:

File "/app/pypowerwall/init.py", line 236, in poll payload = self.client.poll(api, force, recursive, raw) File "/app/pypowerwall/fleetapi/pypowerwall_fleetapi.py", line 490, in get_vitals config = self.fleet.get_site_info(force=force) File "/app/pypowerwall/fleetapi/pypowerwall_fleetapi.py", line 183, in poll return func(**kwargs) File "/app/pypowerwall/fleetapi/pypowerwall_fleetapi.py", line 386, in get_api_system_status_soe percentage_charged = self.fleet.battery_level(force=force) or 0 File "/app/pypowerwall/fleetapi/fleetapi.py", line 483, in battery_level return self.keyval(self.get_live_status(force=force), "percentage_charged") File "/app/pypowerwall/fleetapi/fleetapi.py", line 240, in get_live_status payload = self.poll(f"api/1/energy_sites/{self.site_id}/live_status", force=force) File "/app/pypowerwall/fleetapi/fleetapi.py", line 185, in poll "Authorization": "Bearer " + self.access_token File "/app/pypowerwall/fleetapi/fleetapi.py", line 348, in get_site_info payload = self.poll(f"api/1/energy_sites/{self.site_id}/site_info", force=force) TypeError: can only concatenate str (not "NoneType") to str File "/app/pypowerwall/fleetapi/fleetapi.py", line 185, in poll "Authorization": "Bearer " + self.access_token

I need to dig in to that...

jasonacox commented 4 months ago

@mcbirse and @spoonwzd - I renamed aux.py to regex.py (basically all that file contains).

I made some changes to address the token renewal logic failure. 🤞

jasonacox/pypowerwall:0.9.0t56-beta11

jasonacox commented 4 months ago

jasonacox/pypowerwall:0.9.0t56-beta12

emptywee commented 4 months ago

@jasonacox wow, that's a huge PR and great work!

jasonacox commented 4 months ago

Thanks @emptywee !

I haven't see the token renewal issue occur again but I'm going to run it a bit longer before I call it good. If anyone gets a chance to try it out, please let me know.

Powerwall-Dashboard users can test by deploying the jasonacox/pypowerwall:0.9.0t57-beta container:

  1. Edit powerwall.yml and change the image line to jasonacox/pypowerwall:0.9.0t57-beta
  2. Restart with ./compose-dash.sh up -d
  3. Setup FleetAPI with: docker exec -it pypowerwall python3 -m pypowerwall fleetapi - If you haven't done this before, it invovles more than the cloud setup, specifically setting up an account with Tesla (see here)
  4. Restart pypowerwall: docker restart pypowerwall

Check logs with docker logs -f pypowerwall

jasonacox commented 4 months ago

Testing: Running for several days now. A few issues:

The first one seems to be network related and acceptable - We would want the library and proxy to surface those.

The Second one typically means that the request (telegraf in this test) exceeds the timeout periods and disconnects, indicative of pypowerwall having trouble fetching the required data from FleetAPI within an acceptable time. I just realized that the timeout from class PyPowerwallFleetAPI is not passed on or honored by the FleetAPI class. It may not eliminate all of these errors but could help, likely revealing timeouts with the FleetAPI service. An alternative would be to increase the cache timeout, but I'm keeping it at 5s to mirror the PyPowerwallCloud class.

emptywee commented 4 months ago

The Second one typically means that the request (telegraf in this test) exceeds the timeout periods and disconnects, indicative of pypowerwall having trouble fetching the required data from FleetAPI within an acceptable time. I just realized that the timeout from class PyPowerwallFleetAPI is not passed on or honored by the FleetAPI class. It may not eliminate all of these errors but could help, likely revealing timeouts with the FleetAPI service. An alternative would be to increase the cache timeout, but I'm keeping it at 5s to mirror the PyPowerwallCloud class.

I'd say the second one means that the other end of the established connection reset it (aka closed unexpectedly). And depending on where exactly this error shows up in the source code, it may mean different things.

If this is what prints it: https://github.com/jasonacox/pypowerwall/blob/fleetapi/proxy/server.py#L661 It'd mean that by the time self.wfile.write(message.encode("utf8")) attempt is made, the socket has already been closed by the client who connected and requested something. Question is: why did the client disconnected that early? It didn't like the headers the proxy sent? Or generating the response by the proxy took too long so that the client having its own timeouts stopped waiting and disconnected?

jasonacox commented 4 months ago

An issue I have discovered with FleetAPI - The token renewal process requires a "one time use" renewal token. Once that token is used, if you don't get the new token, you are unable to renew without re-running the setup process (requires you to re-auth with Tesla + 2FA). The good news is that the semaphore logic I added to FleetAPI class is working to ensure that token is used only once. However, if you try to use another instance of the proxy, the refresh token will have been used and renewal will fail. This is an edge case, so not terribly concerning, but a different behavior than the Tesla Owners API. I need to look at that again to ensure I'm not missing something that should also apply to FleetAPI.

jasonacox commented 4 months ago

It'd mean that by the time self.wfile.write(message.encode("utf8")) attempt is made, the socket has already been closed by the client who connected and requested something. Question is: why did the client disconnected that early? It didn't like the headers the proxy sent? Or generating the response by the proxy took too long so that the client having its own timeouts stopped waiting and disconnected?

Yes, that is what I have been able to deduce as well. The client in this case is telegraf polling the proxy which has timeout = "4s" which was set to accommodate a 5s sample frequency without layering connections.

emptywee commented 4 months ago

Yes, that is what I have been able to deduce as well. The client in this case is telegraf polling the proxy which has timeout = "4s" which was set to accommodate a 5s sample frequency without layering connections.

Likely, telegraf closes it after 4s which isn't enough for the proxy to query external API, in this case.

jasonacox commented 4 months ago

I noticed the Powerwall capacity data, specifically total_pack_energy and energy_left are no longer showing up in FleetAPI data. I checked and see that it is also not showing up in the Owners API and was identified as a change in March (TeslaPY https://github.com/tdorssers/TeslaPy/issues/161). I spent some time hunting for them but don't see anything.

Tesla FleetAPI documentation shows it should be returned:

image

But I only get this:

pw.client.fleet.poll('/api/1/energy_sites/{site_id}/live_status')
{
    'response': {
        'solar_power': 2320,
        'percentage_charged': 87.8548799182422,
        'backup_capable': True,
        'battery_power': -1080,
        'load_power': 1240,
        'grid_status': 'Active',
        'grid_services_active': False,
        'grid_power': 0,
        'grid_services_power': 0,
        'generator_power': 0,
        'island_status': 'on_grid',
        'storm_mode_active': False,
        'timestamp': '2024-05-20T15: 55: 10-07: 00',
        'wall_connectors': [           
        ]
    }
}

The code is doing what it should and returning None for these data points since they do not exist.

jasonacox commented 4 months ago

I did find a problem with the setup mode. If you set it up already, it was attempting to use the refresh token and if it was already used, 401 response caused the script to fail. If someone is running setup again, the expectation is that you are wanting a new bearer and refresh token.

Updated timeout argument and 5s default to FleetAPI: jasonacox/pypowerwall:0.9.0t57-beta2 jasonacox/pypowerwall:0.9.0t57-beta3

jasonacox commented 4 months ago