Communication with high PCS CPU Load

[fu-zhou]

It seems that on my system (with a high alternating load during the day) the PCS has a lot to do managing the energy flow: charge battery, discharge, get from grid, feed to grid etc. The poll time of 10 seconds regularly leads to very low, unrealistic values for the load or power from the PV modules. Increasing the time to 20 seconds improves the stability of the transmitted values, but with a high load on the PCS's CPU, the communication seems not to be executed with a high priority. This still leads to a few unrealisticly low values being transmitted. There is no need to receive the daily or monthly accumulated values every 10 seconds, but for instance the load, PV power and active grid power would be good to have reliably every 5 seconds or even faster (I control the load with those values). Is there a way to define multiple intervall_times by e.g. starting two instances of essmqtt with limited data points each? Unfortunately the PCS's CPU load isn't part of the protocol and I have no idea if transmitting a few values with 5 seconds and the rest with 60 seconds would be handled reliably by the PCS. I would test the scenario but have no clue how to set that up.

[gluap]

Hi @fu-zhou, are you using graphite as well as the mqtt connection? Im asking because graphite connection uses same fast polling interval.

Is the issue especially visible around noon? My observation is that the current from the modules becomes more variable when the battery is full and ess is operating either close to maximum ac load or close to the feed-in limit.

Finally even without pyess running turning on or off large loads (oven, water boiler etc..) i always see a short spike on my power from grid usage or grid feed-in as reported by my smartmeter. I think that the ess takes up to a twoo or three seconds following a load change.

I will see if I can get find a good way of setting the polling times independently.

What it looks like during the day:

[fu-zhou]

Hi @gluap: I'm running MQTT only. Yesterday the issue was around noon when the battery was full and the load partially exceeded the PV power. Similar situation in the afternoon with the dish washer and dryer on, which means I can also follow your observation: full battery, high load, high power from the roof (which points me at a high CPU load, as well). With 10 seconds poll time the spikes look like: And with 20 seconds: The question is: Will longer poll cycles on the majority of values lead to that a handfull of values can be reliably shown in shorter cycles? Once a method of setting multiple poll times is clear, I'll test the heck out of the system!

[gluap]

@fu-zhou I've been thinking about this - I think the CPU load may not be the reason of the spikes after all. First of all there are three different version numbers displayed in the "System information" of the app. These seem to be the controllers for different parts of the system. I would guess that the ESS has separate controllers, dealing with the power distribution (this has to be near realtime so likely it is separate from the networking / API server).

The fact that the spikes don't show on enervu or in the history view of the app is likely because the ess does only store (and upload) a coarse history of the values, much less resolution than a sample every 10 seconds so such a spike is likely to be just smoothed over.

My guess is that you see less spikes with a higher polling time might simply be because you are only generating half as many samples. Therefore it follows that you would also see only about half as many spikes, even if they were randomly distributed and not CPU related. From the two plots this seems still a reasonable assumption.

Since I can't exclude that the spikes are caused by CPU load:

The data is obtained from ESS in 2 requests:

1. https://<ip>/v1/user/essinfo/common for a json response with everything you find below /ess/common in mqtt
2. https://<ip>/v1/user/essinfo/home returns a json response with everything you find below /ess/home in mqtt There is no way to ask for just single values below "home" or "common", the box delivers everything together or nothing.

I see realtime values below both - for instance:

ess/common/PV/pv1_voltage 26.500000
ess/common/BATT/dc_power 190
ess/common/LOAD/load_power 191
ess/home/statistics/batconv_power 190

If anything one could slow down requests for common, on first glance home is smaller and contains most realtime values. Furthermore home is what is refreshed by the app every 10 seconds in the default view so it should really be very safe for polling every 10 seconds. (But on other screens, the app also polls common repeatedly so I wouldn't expect problems w.r.t. this.)

I attached a wheel for you to test - after unzipping you can install it in a venv of your choice by running pip install /path/to/pyess-0.1.3b0-py3-none-any.whl.zip pyess-0.1.3b0-py3-none-any.whl.zip

essmqtt in the wheel has an extra command line parameter --common_divisor that makes it poll common only every nth poll of home. So with the default 10 second polling and --common_divisor=6, common will be polled every 60 seconds. The argument to --common_divisor is expected to be an integer.

[fu-zhou]

@gluap : I got the test version installed, but the divisor doesn't work. essmqtt understands --common_divisor=6, but the server still updates the common values every 10 seconds.

I compared the "common" and the "home" section and found the following redundancies:

mqtt.0.ess.common.BATT.dc_power = mqtt.0.ess.home.statistics.batconv_power
mqtt.0.ess.common.BATT.soc      = mqtt.0.ess.home.statistics.bat_user_soc
mqtt.0.ess.common.BATT.status   = mqtt.0.ess.home.statistics.bat_status

mqtt.0.ess.common.GRID.active_power = mqtt.0.ess.home.statistics.grid_power
# grid power shows positive values only => feed in can not be recognized by value

mqtt.0.ess.common.LOAD.load_power = mqtt.0.ess.home.statistics.load_power

mqtt.0.ess.common.PV.pv1_power +
mqtt.0.ess.common.PV.pv2_power +
mqtt.0.ess.common.PV.pv3_power = mqtt.0.ess.home.statistics.pcs_pv_total_power

In terms of updates and accuracy/ reliability the most important value for me is the grid power incl. correct positive or negative sign. The sign is being provided only in the common section, but in "home" there is mqtt.0.ess.home.direction.is_gridselling. If "1" (=selling to grid), mqtt.0.ess.common.GRID.active_power is negative according to the records: Which means mqtt.0.ess.home.direction.is_gridselling can be used to multiply mqtt.0.ess.home.statistics.grid_power with -1. I have to try that, but most important is to get the common_divisor working...

[gluap]

I found my mistake -- I added the argument and also implemented the functionality but failed to carry the value from the command line through the function. Here's another zipped wheel that should work now.

pyess-0.1.4b1-py3-none-any.whl.zip

Yes, these were the redundancies I meant - Basically home is most of what one needs, common is more extensive and repeats some of the values from home. In exchange it also contains a lot of cumulative values that only update every 15 minutes or so. So faced with the question which of the two to collect less frequently I think common is the intuitive choice. edit: If you disagree it wouldn't be an issue to make home the less-often-read data.

One thing you might check is whether you see the spikes in both the values from common and home or whether only one of the two is affected. And if they occur in both if whether they occur at the same time.

edit to add: thanks for pointing me towards the fact that the GRID contains the grid power with a proper sign. So far I had used the method combining the direction with the value from home.

[fu-zhou]

Okay, the common_divisor works now and I'm crazy enough to try 5 seconds for home and 60 seconds for common with a divisor of 12.

I fully agree that it makes more sense to retrieve the home values in a faster cycle than the common values for the already mentioned reasons.

Regarding the spikes in the redundant values of home and common:

• I refrained from recording the redundant values in home, but I started that now for the two values grid_power and load_power.
• Since common is currently being polled every 60 seconds only, we'll see how the spikes behave, however I plan to set the divisor to 1 tomorrow or so in order to see how the spikes behave in the two sections with the same poll time.

So far, so good, thanks for all you effort!

[fu-zhou]

Hi @gluap, while on a short break now, two observations looking at the curves from last night (limited to the resolution of 5 resp. 60 seconds): 1.

• Load Power (pink) jumps from +440 W to +531 W
• Grid Power (orange) jumps from +70 W to +831 W (=buying from grid - while I don't understand the leap by 760 W with the load leaping by 90 W only)
• Battery (brown) reacts with a jump from +376 W to +1247 W (discharging) which results in "selling to grid" (green) and "discharging to grid" (blue) in the next cycle
• Grid Power (orange) drops to -184 W (=selling to grid, I multiplied with -1 therefore it can be negative even that I took the "home" value)
• Battery supplies power to the grid for a short period of time (reflected in "selling/ discharging to grid")
• System stable again with: load +531 W, Grid +70 W, Battery +461 W (=531 W - 70 W)

This behaviour (with different values at different times) can be reproduced and it seems that the system predicts/ sets somewhat: "Battery-power-increase" = "Grid-power-increase" + "Load-power-increase" resulting in selling power from the battery for a really short time as the PCS controller probably needs to ramp down the supplied battery power once the system is stable again. For my application that means that I need to filter out the "selling to grid" spikes.

2. The common values with a poll time of 60 seconds (the darker lines: black, purple, red) also recognize the spikes as long as they last long enough and it seems that they show like an avarage of the previous 60 seconds.

Of course all this has nothing to do with the communication you tap into and cannot be changed by pyess. But I really like the interval and divisor function as it reduces load on the communication (and my database). By better understanding how the system works you may even come up with additional parameters...

Always ready to keep on testing!

[fu-zhou]

Delaying "mqtt.0.ess.home.direction.is_gridselling" by 10 seconds seems to work well. "is_grid_selling" needs to be "1" for at least 10 seconds before I multiply "mqtt.0.ess.home.statistics.grid_power" with "-1". With this I got rid of the "is_grid_selling"-spikes in my application which lead to briefly turning on a load ("grid selling" means excess energy which can be used rather than sold for 9.5 Cents/kWh). A single spike on the top line (green) doesn't turn the grid power to a negative value. Okay, with weather like today it is no fun anyways and I probably have to run a couple of more tests... The fed in energy (mqtt.0.ess.home.statistics.grid_power multiplied by -1 with a 10 seconds delay) is the orange line at the bottom while the unmanipulated mqtt.0.ess.home.statistics.grid_power is the greenish line on top of the orange line.

[fu-zhou]

I turned off "Hochladen von Daten zum Web-Server" and that also seems to help to reduce the spikes. I currently run "home" and "common" at 2 seconds to try a couple of things... Update this morning: The PV String currents show 15-minute spikes, not at the same time though...

[fu-zhou]

I have the system running now for a couple of days in 2 seconds poll cycles for both "home" and "common" which works well. The PV-spikes are filtered out the way I display the data now - they are still in the database, but I don't have to see them...

One reason for the spikes could be that they are there in reality, not caused by the lag of communication, but because the MPP trackers are testing where the point of maximum power currently is. An option could be that this happens every 15 minutes and takes a couple of seconds. The 3 MPP trackers don't do this at the same exact time in order to prevent the PV power to go to 0. That would also explain the picture above from 3 days ago. If in my case string 1 (5 kWp) is tested during noon by the MPP tracker, that results in a power drop of 4 kW for the duration of the test cycle, which activates the battery to supply power in case the load is accordingly high. Curves displayed without and with spikes:

[gluap]

@fu-zhou Thank you for your extensive reports. I've been following closely what you reported and your findings shed some light on the possible cause of the spikes.

I agree with your finding - The striking regularity of the spikes that becomes visible with 2 second tracking resolution seems to point towards some scheduled source. Maximum Power Point Tracking would be a likely candidate. Also if CPU load on the ESS were indeed the source of the issue as initially suspected I would expect it to become much worse with two second intervals. I can't really imagine how CPU load would lead to 15-minute-spaced spikes with constant time offsets between strings.

MPPT might also explain why I don't see the spikes: My modules have power optimizers installed because I have a suboptimal roof with modules in a string suffering inhomogeneous shading throughout the day. This might be the cause for my ESS to behave differently, possibly MPPT is disabled in my ESS (I am unsure whether ESS is power optimizer-aware though).

Once you consider the research finished I'd suggest to close the issue. I'll leave it open for some time in case you want to add something.