Sensors for consumed heating & consumed water heating subtract energy usage every night

[Schlichi83]

I have a graph (apex chart) for my consumed energy for heating and water heating using. For about a week now the chart doesn't work anymore properly.

I use the following sensors: sensor.stiebel_eltron_isg_consumed_heating_today sensor.stiebel_eltron_isg_consumed_water_heating_today

While the day is still running, everything is okay - but the next day the value for the previous day is reset to 0. I checked the statistics data and saw that every day at 23:55 the energy usage for the whole day is subtracted - see screenshow below.

If I correct the statistic value and set this to 0 everything works again. But this isn't a proper solution to manually correct this every day. Is this intentional? Because up to a week ago everything worked for almost a year. I know I could use the total sensors, but then I'd be missing today's values.

[Padanian]

The total of the day is reset around midnight. It seems that this is an issue pertaining to home assistant, more than to the integration. It should save the data around 23.45, instead than at midnight.

[Schlichi83]

Okay, if this is the case - how can I bring this up to the Home Assistant developers?

[starlight2k]

After upgrading to 2024.02 it looks like there was a change in the energy board, maybe related to breaking the table of the energy dashboard because of consumers bars getting too long ? now the table looks fine on any device, but consumer bars are negative

[CSchlipp]

I am having the same issues since some days. Looking through the recent changes, both on Homeassistant and here, my guess is that https://github.com/pail23/stiebel_eltron_isg_component/pull/177 broke the energy integration.

The way I understand https://developers.home-assistant.io/docs/core/entity/sensor/#available-state-classes total_increasing was the right state class, where a reset to 0 close to midnight is considered a meter reset and not just a decreasing value like reported above.

[Padanian]

will you try and see whether that would solve the issue?

[CSchlipp]

I have reverted to 2024.1.1 to check over the weekend.

[CSchlipp]

In version 2024.1.1 the energy values are reported correctly. Could you please revert #177?

[CSchlipp]

This is just the new hourly graph so far. As your graph is still showing the negative values at the beginning, it's the same wrong behaviour as reported earlier. The actual error can only be observed on the next day, when the value of the previous day is reset (which it shouldnt)

[Padanian]

Yeah, needs to wait tomorrow. I browsed the changelog of 2024.3 and there's nothing that could have affected the total. I hope Paul releases soon the fix I prepared few days ago.

[pail23]

The xy_today energy sensors are not suitable for the energy dashboard since the counter resets every day to 0. I have added in the release 2024.2.2 new energy sensors (e.g. Produced heating). They should work fine in the energy dashboard. Please test.

[kucerazdenek]

Then we will loose the history in energy panel I am afraid?

[CSchlipp]

I have multiple other integrations/meter that report a daily value and reset around midnight. All of them are working fine with the energy dashboard. Daily sensors do have a value besides total sensors, and I would appreciate them being supported in future as well.

While the new sensors (just judging by the code) will work with the energy dashboard (also total values are properly split by homeassistant into hourly/daily shares), the daily sensors are still technically wrong. create_daily_energy_entity_description should report state_class=SensorStateClass.TOTAL_INCREASING as well. TOTAL_INCREASING sensors are allowed to be reset as well, indicating a new meter cycle (=a new day). Ref. https://developers.home-assistant.io/docs/core/entity/sensor/#available-state-classes:

Similar to total, with the restriction that the state represents a monotonically increasing positive total which periodically restarts counting from 0, e.g. a daily amount of consumed gas, weekly water consumption or lifetime energy consumption. Statistics of the accumulated growth of the sensor's value since it was first added is updated every 5 minutes. A decreasing value is interpreted as the start of a new meter cycle or the replacement of the meter.

The old implementation before #177 was reporting the correct values, reset at midnight, and the values in the energy dashboards of the past days were still correct. After #177 the meter value is just going up and down, without HA considering it as a "reset", leading to the weird negative bars reported earlier.

https://github.com/pail23/stiebel_eltron_isg_component/issues/153 was caused by the value for whatever reason changing to a lower value than before, which should be caught by the integration. Strictly speaking, it still isn't, the issue was just circumvented by switching to the wrong state_class 🤷🏼‍♂️

[Padanian]

IMHO a monotonically increasing positive total cannot go back to zero. We should notify home assistant that the description is misleading. In fact, #153 error was caused by home assistant correctly catching a decreasing value for a monotonically increasing variable. If we revert #177 we will be back to square one with the same error.

[CSchlipp]

The error message in #153 was complaining about a non-zero value, that was lower than the previous one (from 68 to 67). In case a TOTAL_INCREASING meter is reset to zero, there is no exception. I just double checked that with 2024.1.1 after midnight.

177 never actually fixed #153, to do so the Integration would need to explicitly check if a reported value is greater or equal than the previous one or 0 (for the reset), and otherwise discard the invalid reading.

In Pseudocode: if ((newValue >= oldValue) || (newValue == 0)) then send(newValue) endif

[kucerazdenek]

@pail23 , @Padanian , just tested updated version of integration 2024.2.2 and new cumulative sensors. Unfortunately those don't work by design. Sensor stiebel_eltron_isg_consumed_heating_today is reset to 0 around 11:55-11:58 pm (before midnight), while stiebel_eltron_isg_consumed_heating_total is updated after midnight by ISG. This leads to the subtraction of the daily consumption from the current day first (negative graph) and then to the addition of this consumption at the start of the following day (positive graph with consumption equal to previous day for first hour) as shown on pictures below. I would intercede to revert #177 as proposed by @CSchlipp, because daily sensors worked fine till this change. Discussion about meaning of monotonically increasing positive is just formal, I believe that primary objective is to have working solution. Also, if you plan to remove daily consumption sensors as mentioned in release notes, energy graphs for more than year (in my case) will get useless then.

I have to mention that I really appreciate all the hard work you do in your spare time and for free, so that BFU like me can integrate a heat pump into HA and manage my household.

New cumulated sensor stiebel_eltron_isg_consumed_heating

End of previous day, new cumulative and original daily sensor in negative

Start of new day, cumulative sensor shows positive value with amount of energy consumed during previous day

[kucerazdenek]

OK, because there is no response since last week, I have changed the below in my sensor.py. No subtraction of daily sensors at the end of the day anymore. Please fix it in this matter in next release. Thanks a lot.

state_class=SensorStateClass.TOTAL,

to

state_class=SensorStateClass.TOTAL_INCREASING,

def create_daily_energy_entity_description(name, key, visible_default=True):
    """Create an entry description for a energy sensor."""
    return SensorEntityDescription(
        key,
        name=name,
        native_unit_of_measurement=UnitOfEnergy.KILO_WATT_HOUR,
        icon="mdi:meter-electric",
        has_entity_name=True,
        state_class=SensorStateClass.TOTAL_INCREASING,
        device_class=SensorDeviceClass.ENERGY,
        entity_registry_visible_default=visible_default,
    )

[CSchlipp]

As there was some hesitation on using SensorStateClass.TOTAL_INCREASING for the energy sensors, #222 introduces the last_reset property. Whenever a meter resets to 0, the last_reset property will be set. This tells HA that the meter value hasn't just decreased, but that a new meter cycle has started. There will be no negative reading in the energy dashboards with this change. Decreasing values as reported in #153 will be considered correctly.

Additionally, the misbehaving combined sensors have been removed, making it backwards compatible (including historic values) to releases <= 2024.2.1 again, as noted/reported by @kucerazdenek.

[CSchlipp]

I tested the latest changes over the past days. 1) The reset of the daily sensors is correctly displayed by HA now, no more negative bars in the energy view. 👍🏼 2) The improved implementation of the (daily + total)-sensors is working as well, no decrease & increase at the end of the day anymore. 👍🏼

While comparing both values, I noticed something interesting though: This difference between daily and total sensors can also be observed in older versions, it seems like the daily sensor doesn't get the last update of the day. I guess this is a limitation from the WPM?

[starlight2k]

• the problem was already described somewhere... you can only read full kwh with modbus
• isg webinterface shows three decimals

what ive seen: the values are not live and only refresh hourly (?)... so it wont get the update for the last day hour, because it resets before midnight

the shown kwh are 10-15% less than real consumption. i installed a shelly 3em for comparison

[CSchlipp]

The saga continues 🤯 The new filter for "only increased values" added by @pail23 has some unwanted side-effects as well. Yesterday, the reported total value had a short spike, presumably while the value was transitioning from the low- to the high-register (997 -> 1001 kWh) Version 2024.2.1: Version "Master-Branch": The new filter kicked in and didn't decrease the value again, making it look like it consumed about 1000kWh more than it did: This wrong filtering affects the total value reported by the WPM as well as the calcualted sum (today + total).

[Padanian]

It is apparent that the storage of canbus values into modbus registers isn't working as expected. I would suggest to refrain using hourly data and scrap it altogether and just use daily cumulative data.

[pail23]

I have reverted the changes to the xy_today and xy_total sensors. In addition, I have improved the spike detection logic for the following sensors:

• Produced Heating
• Produced Water Heating
• Consumed Heating
• Consumed Water Heating

Eventually these sensor might replace the xy_total and xy_today sensor.

Please provide me feedback if the sensors above are properly working for you.