Zonneplan API unstable; cache results (longer)?

[nl0pvm]

Hi,

I am assuming that i am not the only one having the issue below that the zonneplan API has become unstable, probably because of their succes. In economic terms this is called the winners curse.

I hope that it is possible to cache the results as now all automations are all over the place because suddenly all kinds of sensors like the current tariff group are unavailable for short periods.

2023-02-04 15:24:38.910 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/summary')
2023-02-04 16:01:31.294 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/gas')
2023-02-04 16:07:44.764 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/user-accounts/me')
2023-02-04 16:28:03.373 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/gas')
2023-02-04 18:38:08.960 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/electricity-delivered')
2023-02-04 18:50:18.628 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/user-accounts/me')
2023-02-04 19:41:04.782 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/electricity-delivered')
2023-02-04 20:40:15.874 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/summary')
2023-02-04 21:04:51.432 ERROR (MainThread) [custom_components.zonneplan_one.coordinator] Error requesting zonneplan_one data: 503, message='Service Unavailable', url=URL('https://app-api.zonneplan.nl/connections/xxxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx/gas')

[fsaris]

Looks like a temporary hickup on there side

[JohanEikelenboom]

I have the same error, since last friday.

[mvzut]

I have the same, the service is unavailable very often, many times per hour. The own Zonneplan app seems to work fine during those moments. Can it be something else then their server having problems?

[cascer1]

I'm experiencing this issue as well. I think a quick solution could be to simply not update sensors when getting an error like this. Then, in the future perhaps it could be added to expire values that are time bound (like setting the electricity price to unknown when the next hour starts)

[mvzut]

I am also relying on this plugin for my energy use logging. But I'm afraid, when these problems continue, I can better use a P1 splitter and log energy usage directly from the P1 port again, like I did before...

[FreddyX078]

Same here, it is completely unusable at this moment. Goes down every 5-15 minutes.

[JohanEikelenboom]

It looks like the API is working again

[fsaris]

mmm, need to check the HA inner code again. My logs show that even when the api call fails it uses the previous values, but somehow the values are not visible in the stats.

[FreddyX078]

I think they fix it before they left the office because no more errors after 17.00 :-)

[Ascathon]

It seems their app wont work during these times either.

[sniperxnl]

Same issue here, app is working

[hbeekhuis]

Same here, all entities (except gas price for some reason) unavailable right now. A couple of days ago it was the other way around, then the gas price was unavailable.

Where can I find logs, would that help?

[rdgout]

@fsaris we can see in the api that some installations are performing an update request every minute. Our data only refreshes every 5 minutes. So this means that these installations are basically spamming our API with pointless requests. It would be great if you could setup a refresh time of 5 minutes or more. This would reduce the, pointless, load on our api.

[hmmbob]

It was set to 2mins actually, but I've raised a PR to change it to 300sec (=5min)

[hbeekhuis]

As I (and perhaps others?) am only using the hourly changing pricefeeds for my total cost and smart device integrations, perhaps in a future update the interval could be settable according to usage, during setup? So

• when using 'live' data like current power consumption, set 5 min interval
• When using hourly rates only: 30 minute interval (or so)

[deqw]

@hbeekhuis I'm using the price to trigger charging. Your solution results in waiting much too long before I can start and stop charging.

[hmmbob]

It probably needs a redesign of the solution. The price information (the source) actually changes only once a day (around 1-2pm currently) - all info is then "known" and can be used for the sensors later in the day (it won't change anymore - all prices for all timeframes are known already) including the "actual tariff" sensor. So that would negate the need to request all data every 5 mins., we could just update the sensors in accordance with the known tariff at the known timeslot. So your sensors still update in time, but the info is request way less often.

Basically i'm saying "we should be caching" - but with a lot of words

edit: and for some expectation management: I can hack some existing code (hence the previous PRs for some small fixes) but I am no Python dev - so don't hold your breath waiting for me to raise a PR for this 😉

[deqw]

@hmmbob That would work. And maybe doing regular requests when the time window that lays ahead shortens too much (eg. when no updates came in at the expected time).

[fsaris]

Another option would be respecting the etag caching headers. But didn't yet find a native python solution for this. Will give it some more thought. Help is welcome :)

[fsaris]

@deqw @hmmbob I added the ETag to the request header. Now Zonneplan can tell us if there is updated data. Every endpoint is polled every 10 seconds and depending on the ETag check (that they perform) we get new data or a status 304 Not Modified. This should give more up-to-date data for the charge points without hammering there API

[rdgout]

@deqw @hmmbob I added the ETag to the request header. Now Zonneplan can tell us if there is updated data. Every endpoint is polled every 10 seconds and depending on the ETag check (that they perform) we get new data or a status 304 Not Modified. This should give more up-to-date data for the charge points without hammering there API

Hi @fsaris,

This is not how ETags work, an ETag is a generated hash based on the content the server produces. This means that the server still has to do all the work, the only thing it saves is data transfer. So if you move to polling every 10 seconds, our server will be overloaded with a ton of pointless requests.

Please go back to 300 seconds. 😄

What was said before is correct, this plugin should be caching data itself like the prices which do not change for hours.

[fsaris]

@rdgout okay, that's a unfortunate 😄

Will revert it back. Can you tell me with what interval the charge point endpoint updates its values? Especially after performing an action as start/stop charge.

[rdgout]

@rdgout okay, that's a unfortunate 😄

Will revert it back. Can you tell me with what interval the charge point endpoint updates its values? Especially after performing an action as start/stop charge.

Charge points update on a fixed 5 minute schedule, however, they may also send updates every time something changes. What we do in the app is poll for updates after we request a change for a fixed period until we see the change has happened. This will require some sort of keeping state to know when the change has happened otherwise it will be just asking for data continuously.

So basically what you could do:

1. Send instruction (like boost, stop charging)
2. Poll every 10 seconds to see if the change has happened. (started or stopped charging with a maximum of a minute, so 6 times)
3. If the change has not happened after this minute, assume the instruction has failed.

It would be great however if you would not implement this in a naive way where the plugin just keeps trying to send instructions and spam our API anyway, instructions may fail for any number of reasons like:

• The charge point is disconnected (literally turned off by the client)
• 4G is not working due to a network / ISP error
• The charge point is not connected to a car

If you could make sure instructions are sent only when the car is connected AND not in an error state, that would probably resolve most issues that could arise otherwise.

[rdgout]

@fsaris We have noticed that a fair amount of HomeAssistant installations have started requesting updates every 10 seconds, for some installations this has lead to an average of 1 request every second. Did you change something or do these people have a version where the update timer was changed to 10 seconds?

We would love to leave the current limits in place but if this continues we will be forced to implement rate limiting since we are not expecting this many requests.

I also noticed that if a customer has electricity and gas that the integration requests the /summary endpoint for both connections. The data in /summary is essentially the same for each connection.

Can you look into this and see why it's happening?

[fsaris]

@rdgout the refresh was only for a short time set to 10 seconds and this wasn't part of a official release. Maybe it's the charge point users that are testing a beta, but there is already a new release that only polls every 5 minutes.

Will have a look at the summary call

[fsaris]

@rdgout the 0.021b1 release had a 10sec interval for the poll. Just removed that beta release. The new beta release was released yesterday so all users that use the beta will probably update soon and that should result in less traffic for your server's