Understand and work around for the 0 W reponses for Solarbank 2 and smartmeter

[thomluther]

Solarbank 2 changed the data update frequency to the cloud. For Solarbank 1 it was min 60 sec update interval. The same interval is used by the App to refresh the data on the home screen.

Solarbank 2 home screen update data much more often. Testing needs to be done by the community to answer following questions:

1. What is the min. interval the data can change on home screen?
2. When using the solarbank monitor with a short query interval for the sites, what is the maximum lag of the Data timestamp when solarbank data was sent to the cloud? Need to understand how often this is updated in the cloud.
3. When there are False 0W values returned, how long does that condition exist?

Following is my assumption on the cloud changes for Solarbank 2. The devices can now send data much more often to the cloud. Lets assume every 3-5 seconds. The cloud needs a while to receive and provide updated data upon queries from the App. I assume however this is just a short period for 1-2 seconds only. When the App however queries the cloud data in such an update interval, the cloud response indicates with the new display_data field whether the data is valid or not to the app. In this case, the App does not refresh the home page and skips it for the next request.

So it depends on how often the condition exists in the cloud, how long it is, and how often the data is queried by clients. In worst casees the client always hits such an update period of the cloud and receives false data for multiple queries.

If that is the case, a simple work around can be implemented in the update sites method to re-query the cloud up to 5 times with 1 sec. delay until a valid response is received for the update_sites run. Since the HA Integration uses slower intervals (min 30 sec) to limit the number of Api requests around the clock, each update_sites run should ensure that valid data can be provided. In worst case when all retry attempts fail, the HA integration will need to make use of the false display data and render the dependent entities to unknown until valid data is provided again on subsequent intervals.

This may break existing automations and scripts if not coded properly to expect unavailable values when calculating with number entities.

Skipping the false data without entity update is not a recommended approach, since that will cause stale and wrong data tracking in HA for unknown periods, without having an evidence in history whether data is valid or not.

[thomluther]

I updated the solarbank monitor tool and added support for the smartmeter info structure, so it will now be created as separate device in the Api devices cache. The solarbank monitor will show it and also a new metrics for valid device data. If Solarbank display data is false, data for the solarbank and smartmeter devices are considered invalid, likewise the whole scene_info data in the system cache. Update_sites routine will now requery up to 5 times when invalid response data is received. This is shown in the solarbank monitor: Cloud-Updated : 2024-06-22 16:37:52 Valid Data : YES (Requeries: 0)

This should be usable to track in realtime whether 5 retries are typically sufficient to ensure valid data per query interval. The monitor refresh period can also be lowered to 5 sec to compare the query time with the cloud update time and identify how much the cloud data can age. This will give a hint about the cloud update interval of Solarbank 2.

[lumeenous]

I noticed that in Home Assistant the 0 W responses occur for approx. 5 min, then one real response is transmitted and so on. However, when I open the Anker app with my main account on my phone, the 0W responses in Home Assistant are gone. There are just real values until approx. 1 min after the phone app is closed. Then the 0 W responses starting again. I assume the App sends a kind of request to the solarbank 2, to send data much more often (every second or so) as the momentary values in the app are changing almost in realtime after they are initially loaded. Im afraid I can't specify my observations further, as I have not enough programming knowledge to run the python API myself, but hope it will give some hints for the right direction.

[nicolinuxfr]

However, when I open the Anker app with my main account on my phone, the 0W responses in Home Assistant are gone.

I have noted the same thing, anecdotally too. If I keep the Anker app open on my phone, I never have 0 values. After the app goes to the background, I start to have 0 values again.

[thomluther]

I assume the App sends a kind of request to the solarbank 2, to send data much more often (every second or so) as the momentary values in the app are changing almost in realtime after they are initially loaded.

If that is really the case, then we have an issue here with the Api since it cannot trigger the Solarbank 2 to send continous data. If possible, please test the latest solarbank monitor tool, it utilizes the requery capability if there is invalid data. You can use the shared account for the power value and response tracking, and an increase in retries count will show if that helped to get a valid response up to 5 seconds after invalid data was received. If that does not help to reduce the invalid responses while the App is closed, I have no idea how they could be avoided for a query interval. Regular Api request via the integration will then always show invalid/unusable data at indefinite intervals like HA does show it now.

[nicolinuxfr]

If possible, please test the latest solarbank monitor tool, it utilizes the requery capability if there is invalid data.

How should we proceed exactly ? Wait until the data disappears inside Home Assistant and then use it ?

EDIT : re-reading the README (sorry), I think you mean to use the monitor tool instead of export one. So I will try it now.

EDIT2 :

I have this error, do you need a bug report or did I do something wrong ? I pulled the project this morning.

 File "/home/nicolas/anker-solix-api/./solarbank_monitor.py", line 181
    f"{'Cloud-Updated':<{col1}}: {update_time:<{col2}} {'Valid Data':<{col3}}: {'YES' if dev.get('data_valid') else 'NO'} (Requeries: {site.get("requeries")})"
                                                                                                                                                 ^^^^^^^^^
SyntaxError: f-string: unmatched '('

[thomluther]

I fixed that now, used the wrong quotes in fstring. When you see it in HA showing 0, you can start the monitor tool and watch whether this received data. If you can use a different account for the monitor tool, you can leave it up running. I would assume, that both accounts get same responses for the Scene Info query, so either both are valid or invalid, except the invalid state is just very short and both query at different times.

[thomluther]

Another assumption how Solarbank 2 provides data to cloud:

• When App is starting the homepage, the solarbank is triggered to send data very often (every 3 sec?). This would be similar behavior as watching the device details real time data, where the Solarbank is triggered to send every second to the MQTT cloud server when history is watched in App. (Therefore the device history always starts from empty diagramm)
• When App is not watching the home page, they solarbank provides cloud updates only every 5 min? This could have been implemented to:
  • reduce Api traffic for better scaling with more devices
  • Still sufficient to do all the energy statistics in the cloud (Energy is not done by devices).
• The sent 5 min refresh is just valid data for 1-2 min, then responses indicate that data is state with the new is_display_data field?

However, just receiving a new Watt value every 5 minutes seems to be too less to me to do a proper energy calculation. The energy values could be way off from the reality if there is much fluctuation.

[nicolinuxfr]

So, I may have burnt my Anker account really quickly… 😬

(100053) Anker Api Error: Please try again in 24 hours.
Response Text: {"code":100053,"msg":"Please try again in 24 hours.","trace_id":"29b5a340169fa5a76353abcf0ac765c7"}
<class 'api.errors.RetryExceeded'>: (100053) Anker Api Error: Please try again in 24 hours.

Before that, I had no value at all using the monitoring tool :

I did not open the app before trying this and I had only 0 in HA.

Weirdly (?) just after that, I opened the app on my phone and the data appeared as usual in HA, knowing that I'm using the same account here for the API and inside HA.

[thomluther]

every time you switch an account to another client/device it has to re-authenticate. You can do that up to about 10 times a day, then you get an IP ban. You can restart internet connection on your router to get a new IP, then it will work again.

[thomluther]

BTW, when you use the same account in HA and in the solarbank monitor, they kick out each other all the time. After 10 times there is the IP ban. When you want to use the HA account in the monitor, you need to disable the Api usage first in the Solix system entity. Then it will consume just 2 re-authentications when you switch forth and back between the monitor tool and HA

[thomluther]

Also visible from the screenshot:

• The requery work around does not resolve the problem
• 5 sec refresh interval may trigger too many Api requests per minute. So 10 sec interval is probably better

That means, at this point, I have no idea how to resolve the data collection issue with Solarbank 2. That could mean we cannot really support it. Or you would see unavailable entities with just a single value about every 5 minutes.

[nicolinuxfr]

BTW, when you use the same account in HA and in the solarbank monitor, they kick out each other all the time. After 10 times there is the IP ban. When you want to use the HA account in the monitor, you need to disable the Api usage first in the Solix system entity. Then it will consume just 2 re-authentications when you switch forth and back between the monitor tool and HA

I forgot about that, indeed.

every time you switch an account to another client/device it has to re-authenticate. You can do that up to about 10 times a day, then you get an IP ban. You can restart internet connection on your router to get a new IP, then it will work again.

I only have one fixed IP, but it's a good general tip for others. As for me, I will try again tomorrow if needed.

[thomluther]

Also visible from the screenshot:

* The requery work around does not resolve the problem

* 5 sec refresh interval may trigger too many Api requests per minute. So 10 sec interval is probably better

That means, at this point, I have no idea how to resolve the data collection issue with Solarbank 2. That could mean we cannot really support it. Or you would see unavailable entities with just a single value about every 5 minutes.

I made another change to the Api:

• Requery up to 3 times with 5 sec delay

This gives longer retry timespan from 5 => 15 seconds, and reduces also the requests per minute to avoid hitting the Api error.

[thomluther]

From what I have red so far, there is no way to get around the 0 values issue. It appears Anker has completely changed the mechanism how Solarbank 2 is providing data to the api cloud. That seems to be similar to the real time data transfer to the MQTT cloud, which is triggered and started only when device details are accessed through the app. Likewise this applies now to Api cloud. Per default the Solarbank 2 seems to send Data only about in 5 min intervals. This data will be kept valid in the cloud and then be marked as stale after 60+ seconds if no new data received. Therefore the Api returns valid data only once or twice every 5 minutes interval when the App is not open and triggering the same System for frequent updates. Any immediate requery will not solve the problem, and also shorter or longer intervals will not solve it. When the App is closed, the shortest data refresh intervals will become 5 minutes with Solarbank 2...a major drawback to my opinion. So any history recording or energy calculations from power values will become inaccurate and maybe useless. I assume the App trigger to provide frequent cloud updates is made like for MQTT updates and I have no glue how that is done. Someone needs to reverse engineering the App Traffic, but requests and content will be EDHC encrypted, so not possible to decode without knowing the shared secret from App and Server.

I'm afraid the Solarbank 2 users have to live with that limitation and in worst case this mechanism will be implemented also for Solarbank 1 users, breaking any usefull automation capabilities further. But pretty certainly this change was on purpose to limit cloud data traffic for scale and/or costs.

Question is how the HA integration should deal with invalid responses?

1. Skip entity update until valid data is received?
2. Report unavailable state while data is invalid?

Each approach has pros and cons

[nicolinuxfr]

I think the first approach is better to monitor the system, not to automate it. But if it's not possible otherwise, I would rather have new correct data each 5 minutes than regular useless data. It would not be a big problem for the energy dashboard, for instance.

Maybe I'm missing something here ?

[stephan-l]

I'd prefer valid data too. Right now I just ignore all zero values in my database and print anything else:

Resolution is not really great but still usable for logs. But I made another discovery: if your admin account is not signed in the app the guest accounts don't show data. You just get dashes.

[stephan-l]

To clarify: It only shows data as long as your API shows valid data. So guest accounts seem not triggering an update.

[thomluther]

To clarify: It only shows data as long as your API shows valid data. So guest accounts seem not triggering an update.

That is a very good point. If you can validate, that using only the App with a shared account does show the same issues (empty values except every 5 minutes for a minute or so). you have a capability to officially address this as an issue to Anker. The app must support continous reporting for family members as well, otherwise the member account usage is broken for Solarbank 2 system!

From what I believe, shared accounts have access ONLY to the Cloud Api and the system sharing only occurs there, so a shared account can only query a subset of information from the Api cloud. However, he is not able to use the 'normal' interfaces of the App to the device, since they are assigned a single owner account. That means they probably have no capability to trigger frequent data updates with the device like owner accounts can do.

If Anker will change that to be triggerable by shared accounts via the Api Cloud server, it must be via a normal Rest request that can probably be figured out by someone using tracking apps on the mobile. Other option would be they reduce the standard interval back to 1 minute again and we have the original mechanism as with Solarbank 1, where data do not have to be marked invalid on the cloud. So opening Support tickets with Anker for broken member account usage on Solarbank 2 is the only chance to get this changed to the better by Anker.

For the HA integration there wont't be a situation of reporting valid data only. Either the reported data is stale and potentially incorrect which you will never know (since the are 5 min old or even more). Or the entity become unavailable, so they have NO data instead of wrong 0 values, which can be considered properly by graphing tools and other integrations or templates. Power Integral sensors for instance may be able to sum last known value if source entity is unavailable (not sure however). But summing up an 5 min or older value that is no longer refreshed is not the way the integration should work.

[stephan-l]

I just gave it another try to confirm it. Looks like we are right. I opened a case at Anker to get a feedback.