Loss of Energy Controls - Error fetching Teslemetry Energy Site Info XXXXXX data: Multiple 5xx failures

[purcell-lab]

Checklist

• [X] I have filled out the issue template to the best of my ability.
• [X] This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
• [X] This issue is not a duplicate issue of any previous issues..

Describe the issue

As reported in Discord the Tesla site is returning 500 status errors, which results in the Energy controls going unavailable.

I'm wondering if a delay between calling for live_status & site_info may smooth things out a bit?

Reproduction steps

Config log attached.

config_entry-teslemetry-9148783ba548ef27e70b73108cd5fd97 (9).json

Debug logs

2024-11-08 16:04:29.800 DEBUG (MainThread) [tesla_fleet_api] Requested: https://na.teslemetry.com/api/1/energy_sites/2252097574301386/live_status
2024-11-08 16:04:29.801 DEBUG (MainThread) [tesla_fleet_api] Response Status: 200
2024-11-08 16:04:29.801 DEBUG (MainThread) [tesla_fleet_api] Response JSON: {'response': {'solar_power': 1961, 'percentage_charged': 100, 'battery_power': -200, 'load_power': 6130, 'grid_status': 'Active', 'grid
_power': 4369, 'generator_power': 0, 'island_status': 'on_grid', 'storm_mode_active': False, 'timestamp': '2024-11-08T16:04:29+10:00', 'wall_connectors': [{'din': '1529455-02-E--PGT22325087207', 'wall_connector_
state': 2, 'wall_connector_fault_state': 2, 'wall_connector_power': 0, 'ocpp_status': 1, 'powershare_session_state': 1}, {'din': '1529455-02-E--PGT22325086977', 'vin': '5YJ3F7EC5LF698201', 'wall_connector_state'
: 4, 'wall_connector_fault_state': 2, 'wall_connector_power': 0, 'ocpp_status': 1, 'powershare_session_state': 1}]}}
2024-11-08 16:04:29.802 DEBUG (MainThread) [custom_components.teslemetry] Finished fetching Teslemetry Energy Site Live 2252097574301386 data in 0.672 seconds (success: True)
2024-11-08 16:04:43.126 DEBUG (MainThread) [tesla_fleet_api] Requested: https://na.teslemetry.com/api/1/energy_sites/2252097574301386/site_info
2024-11-08 16:04:43.126 DEBUG (MainThread) [tesla_fleet_api] Response Status: 500
2024-11-08 16:04:43.128 DEBUG (MainThread) [tesla_fleet_api] An error occurred while processing the request.
2024-11-08 16:04:43.129 ERROR (MainThread) [custom_components.teslemetry] Error fetching Teslemetry Energy Site Info 2252097574301386 data: Multiple 5xx failures
2024-11-08 16:04:43.130 DEBUG (MainThread) [custom_components.teslemetry] Finished fetching Teslemetry Energy Site Info 2252097574301386 data in 5.240 seconds (success: False)
2024-11-08 16:04:45.335 DEBUG (MainThread) [custom_components.teslemetry] Streaming received vehicle_data from 5YJ3F7EC5LF698201
2024-11-08 16:04:45.335 DEBUG (MainThread) [custom_components.teslemetry] Manually updated Teslemetry Vehicle 5YJ3F7EC5LF698201 data
2024-11-08 16:04:47.385 DEBUG (MainThread) [custom_components.teslemetry] Streaming received state from LRWYHCFS9PC851970
2024-11-08 16:04:47.385 DEBUG (MainThread) [custom_components.teslemetry] Manually updated Teslemetry Vehicle LRWYHCFS9PC851970 data

[Bre77]

Hey Mark do you have any evidence that other services work when Teslemetry doesnt? I am thinking I'll need to start running a test with Tessie, Teslemetry, and the Fleet API directly to see if there is a correlation.

Here is the number of status 500 messages for energy sites over the last 7 days and the number of users it impacted. A small number of people constantly have issues, but there are also spikes.

[purcell-lab]

Ongoing issue. I mainly use teslemetry for my controls exclusively so haven't noticed the effects on any other platforms.

Is there user ability to reduce or back off the numbers of requests being sent in my behalf?

This error originated from a custom integration.

Logger: custom_components.teslemetry
Source: helpers/update_coordinator.py:414
integration: Teslemetry
First occurred: November 14, 2024 at 7:29:18 PM (77 occurrences)
Last logged: 5:30:43 AM

Error fetching Teslemetry Energy Site Info 2252097574301386 data: Multiple 5xx failures

[purcell-lab]

This isdue is only for energy sites, so Tessie is not relevant.

Here are some comparisons, both are going unavailable at similar times.

Entity display from Teslemetry:

Entity display from Tesla Fleet:

[Bre77]

Hey Mark,

Updates are run every 30 seconds, you can take control of this as such:

On the Integration entries page, you can click the three dots besides your Teslemetry config entry and select system options. Here you can disable polling for updates. You would then need to setup your own automation to update an entity from each coordinator on whatever schedule you want.

There are three energy coordinators (info, live and history) so if you need help identifying a entity from each let me know.

I am now running a test between Teslemetry, Tessie, and Tesla Fleet API directly to try and identify if I can replicate your issue and isolated it to Teslemetry or not. (Tessie does support the whole Fleet API including energy)

This is your requests over the last 24 hours:

And this is mine:

You are making more POST requests to change things than me, so I may need to start changing things randomly to see if that contributes.

I will also turn up the logging on 5xx errors to see if we can learn something new.

[Bre77]

Its probably no help, but you have more errors on the energy APIs than any other Teslemetry user. Today you have been 45% of all 5xx errors on the energy APIs. Every single one, was a response from Tesla saying upstream internal error.

My only two theories are: there is a connectivity issue to your Powerwall, or because you are frequenty changing your configuration, the powerwall itself is unable to respond to API requests at times.

I will continue to monitor.

[purcell-lab]

I do seem to have a lot of devices and entities, but I would image that isn't part of the root cause..:

George (Model Y) was a hire car I haven't had access to for 6 months, but I don't seem to be able to delete..

[Bre77]

George (Model Y) was a hire car I haven't had access to for 6 months, but I don't seem to be able to delete..

Does it still show up in your Tesla app or products endpoint? There is a seperate issue where I need to do device cleanups.

[purcell-lab]

Ok, I have connected my gateway via Ethernet, with 1Gb fibre connection to NBN. Should of done a while ago..

Let's see if this improves connectivity...