MikeBishop commented 2 months ago

It appears that Tesla has now decommissioned the legacy endpoint, as long promised.

09:27:59 ⛽ Manager  20 Charge when above 6A (minAmpsPerTWC).
09:28:13 🚗 TeslaAPI 17 Callandor: stop charge response {'response': None, 'error': 'Tesla Vehicle Command Protocol requ
ired, please refer to the documentation here: https://developer.tesla.com/docs/fleet-api#2023-10-09-rest-api-vehicle-com
mands-endpoint-deprecation-warning', 'error_description': ''}

alexeiw123 commented 2 months ago

While I can't see this in my logs (maybe my log level needs to be changed) I can confirm that TWCManager is no longer starting or stopping charge for me. I will need to work out how to link TWCManager with Tesla Fleet Proxy which I have working in home assistant. I really spent way too much time trying to resolve that over in https://github.com/ngardiner/TWCManager/discussions/570 and ended up giving up once I get the legacy API working again.

MikeBishop commented 2 months ago

What I needed to do to get it working again was:

Get tesla-http-proxy running on the Pi Zero
- Create self-signed certificates for localhost
- Get proxy running in background with those certs plus the private key enrolled with Fleet API
Point TWCManager to the proxy by adding proxy URL, Client ID, and path to self-signed cert to config file

If you Home Assistant plugin is already running the proxy for you, you may only need the second step.

alexeiw123 commented 2 months ago

I think I might have it working. Is there a way to test definitively if the API is working? My doubt is because I can still see this popup:

My setup is rather different in that I use a cloudflare tunnel for remote access to my home assistant, which required some particular settings in order for tesla verification and call backs to work. (more info here if you're curious)

Anyway, what this means for me is that I only have to set the externally available url with valid SSL cert. (of course the clientID and refresh token too).

        "teslaProxy": "https://special.mydomain.url",
        #"teslaProxyCert": "", note theis is commented out

While I make that sound simple, I've gone round and round in circles trying to get this working, largely because of my lack of knowledge in both the Tesla API and how to configure TWCManager properly.

I was seeing in /etc/twcmanager/log/logfile errors like this when I believe I had it set up wrong:

2024-05-12 15:06:01,393 - 🚗 TeslaAPI 20 Could not parse JSON result from https://special.mydomain.url/api/1/vehicles

The good news is those errors seem to be gone but I can't see anything confirming that it's working, and the log-in prompt has me doubting the success.

alexeiw123 commented 2 months ago

Oh, it seems I'm still getting the Could not parse JSON result error.

It seems my refresh token isn't saved from the /settings page unless I put in something the Access Token field, which I didn't think was required with the Fleet API?

If I manually add my refresh token to settings.json "carApiRefreshToken":, then the field gets cleared on me. What should I have for "carApiTokenExpireTime": and "carApiBearerToken": when using fleet API?

MikeBishop commented 2 months ago

TWCManager expects to get:

a bearer token that it can use immediately
- ...and is fresh; expiration is set to 8 hours after you supply it
a refresh token it can use after expiration to get a new bearer token

Technically speaking, if you have a refresh token, you can just use it immediately to produce the bearer token. In one of my other apps (MMM-Powerwall), I actually ignore the user-supplied bearer token if they have a refresh token; I just refresh immediately so I know the token has a full lifetime.

584 makes it read the expiration time from the bearer token instead of assuming. We really ought to just immediately perform a refresh if the bearer token supplied is expired or absent.

Edit: And now #584 mostly does that, too. If the bearer token is already expired or if no bearer token is stored, the refresh token is used immediately to produce a bearer token. What it does that's a bit unexpected: if an existing bearer token is still valid, passing a new refresh token doesn't discard the existing bearer token, it only saves the refresh token for when it decides it needs a new one. Changing that is slighly more involved and might deserve a separate PR.

alexeiw123 commented 2 months ago

Thanks @MikeBishop - I'm still a little unsure on what I should enter when using Fleet API as I don't get a bearer token that I am aware of.

I have generated legacy tokens so these fields are populated, then replaced the refresh token with that from Fleet API.

Then with config.json set up for fleet API I can either provide a selfsigned.pem file, or leave it commented out (as the domain has a valid cert). By doing this I get one of the following two errors.

Using selfsigned.pem:

2024-05-14 11:43:46,714 - 🚗 TeslaAPI 20 Failed to make API call https://my.domain.xyz/api/1/vehicles
2024-05-14 11:43:46,725 - ⛽ Manager  20 BackgroundError: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 841, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 344, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

not specifying a cert:

2024-05-14 11:44:35,038 - 🚗 TeslaAPI 20 Could not parse JSON result from https://my.domain.xyz/api/1/vehicles

Is the 2nd error an indication that the SSL is working but the tesla auth is not? if so that would at least be a step in the right direction.

MikeBishop commented 2 months ago

That seems likely -- it's not complaining about the cert, but it doesn't like the response it gets back. The changes I've got floating in a couple PRs handle getting a 401 back from the API explicitly (by trying to refresh, and failing that, discarding all tokens and logging the condition), but IIRC that's the error I was seeing when TWCManager just tries to continue blithely and relied on parse failing to handle the unauthed condition.

Flitzer42 commented 1 month ago

Now running into the same issue Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 Entering car_api_available - next step is to query Tesla API Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 car_api_available returning True Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 Car API cmd charge_start <Response [403]> Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 17 Model 3: start charge response{'response': None, 'error': 'Tesla Vehicle Command Protocol required, please refer to the documentation here: https://developer.tesla.com/docs/fleet-api#2023-10-09-rest-api-vehicle-commands-endpoint-deprecation-warning', 'error_description': ''} Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 updateCarApiLastErrorTime() called due to Tesla API Error. Updating timestamp from 0 to 1717091082.98671 Mai 30 19:44:56 TWC-Manager python3[500]: 19:44:56 🚗 TeslaAPI 13 Entering car_api_available - next step is to query Tesla API Refreshing the tokens works well this time ;-)

alexeiw123 commented 1 month ago

Now running into the same issue Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 Entering car_api_available - next step is to query Tesla API Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 car_api_available returning True Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 Car API cmd charge_start <Response [403]> Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 17 Model 3: start charge response{'response': None, 'error': 'Tesla Vehicle Command Protocol required, please refer to the documentation here: https://developer.tesla.com/docs/fleet-api#2023-10-09-rest-api-vehicle-commands-endpoint-deprecation-warning', 'error_description': ''} Mai 30 19:44:42 TWC-Manager python3[500]: 19:44:42 🚗 TeslaAPI 13 updateCarApiLastErrorTime() called due to Tesla API Error. Updating timestamp from 0 to 1717091082.98671 Mai 30 19:44:56 TWC-Manager python3[500]: 19:44:56 🚗 TeslaAPI 13 Entering car_api_available - next step is to query Tesla API Refreshing the tokens works well this time ;-)

I don't see this sort of detail in my logs, which would be helpful while I continue to try and get this working. What log level are you using?

Flitzer42 commented 1 month ago

Hi @alexeiw123 , thanks for digging deeper...

Set to

        # INFO9     12        9     includes raw RS-485 msgs tx and rx (2-3 per sec)
        # DEBUG     10       10     all info.
        # DEBUG2     9       11     more than all info.  ;)
        "logLevel": 10,

I did not read any logfiles (failed to handle the access rights to write files....), but used journalctl -f | grep "TeslaAPI to get the info above. 13 could be sufficient, as all lines with TeslaAPI are marked with 13. Mai 30 19:44:37 TWC-Manager python3[500]: 19:44:37 🚗 TeslaAPI 13 startOrStop is set to start Really hope, that you can find a simple to use solution.

alexeiw123 commented 1 month ago

That seems likely -- it's not complaining about the cert, but it doesn't like the response it gets back. The changes I've got floating in a couple PRs handle getting a 401 back from the API explicitly (by trying to refresh, and failing that, discarding all tokens and logging the condition), but IIRC that's the error I was seeing when TWCManager just tries to continue blithely and relied on parse failing to handle the unauthed condition.

I never did manage to get mt TeslaAPI working in TWCManager, piggy backing off of my working FleetAPI in home assistant. Meanwhile, we are about to buy a 2nd tesla and I have a bunch of referral credits going to expire soon, so I've ordered a gen 3 TWC. Unfortunately, the gen 2 and the gen 3 do not load share, so I will likely move to 2x gen 3 units soon.

I think this is probably my time to wave the white flag on trying to use Tesla fleet API in TWC Manager and move to a more 'off the shelf' type solution, such as Tesla's 'charge from solar'

I Just wanted to take a second to thank the devs here that have built and maintained this incredible project, as well as respond to my many noob questions over the years. In particular @ngardiner and @MikeBishop. The running costs on my car over 80,000 km have been next to nothing given the ability for me to charge nearly exclusively from excess solar that I otherwise could not sell.

I'll still be using it for the time being but expect a transition to the lighter cable and load sharing of the gen 3 units.

MikeBishop commented 4 weeks ago

Can't say I haven't considered the same. Currently, at least, I like the ability to have more policy control in TWCManager and running the domain was easy enough for me. But Tesla is deliberately focused on hosted commercial apps, and I don't blame anyone who decides to take an easier path.

alexeiw123 commented 4 weeks ago

I love the flexibility of TWCManager - just losing the ability to start/stop the charge has taken the automation aspect out of it. Seeing as my HA is working, I could set TWCManger to be fixed rate and just do it using automations in HA but I'd struggle to get it to work as smoothly as TWCManager does.

ChutneyMary commented 4 weeks ago

The developer at the link below has made a custom HTTP API as a proxy for making BLE (Bluetooth Low Energy) requests to Tesla vehicles. It installs in a Docker container. The developer notes that no internet access is needed to interact with the vehicle over BLE. The new key paired with the vehicle is seen as a "new car key" for controlling the vehicle.

https://github.com/Jeoffreybauvin/tesla-http-api-over-ble

I had a play with it last weekend, trying to install on a Raspberry Pi Zero W (the same device in my TWC). I had issues trying to build the image but I suspect it might be an issue with the low resources of the Zero W.

I haven't had a chance to start a discussion with the developer, but this might be a way forward?

ngardiner commented 4 weeks ago

Seeing as my HA is working, I could set TWCManger to be fixed rate and just do it using automations in HA but I'd struggle to get it to work as smoothly as TWCManager does.

To be honest, it wouldn't be that hard to call the HA API to start/stop charging. Our problem is that the modular architecture of TWCManager means knowing where to put that code logically is a bit of a challenge and there is also an element of kicking the can down the road, I'd rather just fix API access and everyone would be happy but Tesla have made that difficult at best.

Providing a setting allowing a call to a user defined external API is probably a pretty good way of handling where we are right now. It could be configured to use HA, to Teslemetry, etc with a couple of variables like the URL to call and what parameters to supply.

In practice I am sure it will be much harder than I am making out due to different API approaches but given the multiple threads on this issue all exploring alternative APIs I am wondering if it might be worth a bit of hacking together a proof of concept.

alexeiw123 commented 3 weeks ago

@ngardiner That would actually be very cool. Bonus of that is that is you could very well use HA as a proxy to get charge information from TWC Gen 3 and start/stop/change charging using the HA tesla integration. With no need for RS485 you could run TWCManager on any host.

I'll have both Gen2 and Gen 3 installed in tandem for a bit, so could potentially test that idea.

alexeiw123 commented 3 weeks ago

Although I excused myself - I've now set up my Gen3 on one side of my garage and gen 2 still up and running on the other... and I'm back trying to get TWC working with FleetAPI. Maybe it's the challenge, maybe it's because of the hurdle in getting solar charging working well using HA automation.

Anyway, I'm so close...

Error in my logfile:

2024-06-24 10:28:59,339 - urllib3.co 40 Certificate did not match expected hostname: <hidden>. Certificate: {'subject': ((('commonName', '<hidden>tesla-http-proxy'),),), 'issuer': ((('commonName', 'c03d64a7-tesla-http-proxy'),),), 'version': 3, 'serialNumber': '<hidden>', 'notBefore$2024-06-24 10:28:59,347 - 🚗 TeslaAPI 20 Failed to make API call<hidden>_tesla_http_proxy/api/1/vehicles
2024-06-24 10:29:00,220 - 🚗 TeslaAPI 13 Entering car_api_available - next step is to query Tesla API

Error in my Tesla Proxy add-on log:

2024/06/24 10:28:13 http: TLS handshake error from 192.168.1.5:55318: local error: tls: bad record MAC

It seems that TWCManager is getting through to my Tesla API proxy, but something about the cert isn't right.

@MikeBishop or @ngardiner would either of you be interested in lining up over discord at a time of your convenience and see if we can nut this out with your knowledge and expertise on the subject? I'm just going round in circles without a full understanding of it.

MikeBishop commented 3 weeks ago

I might be able to swing a Discord chat sometime in the next few days. But one thing to try first -- it might be easier to separate the two pieces for debugging purposes. First make sure you can do queries through your proxy, e.g.

curl --cacert tls_cert.pem  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $TOKEN" \
  'https://localhost:4443/api/1/products'

(Obviously, adjusting for location of your proxy and the expected cert file.) If curl doesn't work either, debug the proxy. If it does, make sure TWCManager is finding the same certificate.

alexeiw123 commented 2 weeks ago

I might be able to swing a Discord chat sometime in the next few days

That would be great - thankyou! I had to head off for work for a few days so hadn't given this much attention since then. My discord is .iamlex - hopefully you can find me with that.

But one thing to try first

I already tripped before the starting gate - I don't know how to access my Tesla proxy instance (docker within HA) by console.

ngardiner / TWCManager

"Tesla Vehicle Command Protocol required" #580

584 makes it read the expiration time from the bearer token instead of assuming. We really ought to just immediately perform a refresh if the bearer token supplied is expired or absent.