Unexpected statistics state after upgrade to 2022.2.0

[marc-romu]

The problem

After upgrading to 2022.2.0, all my statistics sensors report an unexpected state. This is a screenshot of one of them:

I didn't change anything from my configuration before nor after upgrading.

What version of Home Assistant Core has the issue?

2022.2.0

What was the last working version of Home Assistant Core?

2021.12.10

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Statistics

Link to integration documentation on our website

https://www.home-assistant.io/integrations/statistics/

Example YAML snippet

- platform: statistics
  name: Temperatura
  entity_id: sensor.temperatura
  state_characteristic: average_linear
  sampling_size: 10000
  precision: 1
  scan_interval: 1800
  max_age:
    days: 1

Anything in the logs that might be useful for us?

<Nothing relevant in logs>

Additional information

Attributes from statistics sensor:

Buffer seems nearly empty, while referenced entity sensor.temperatura is recorded correctly:

[probot-home-assistant[bot]]

statistics documentation statistics source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @fabaff, @thomdietrich, mind taking a look at this issue as it has been labeled with an integration (statistics) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

[marc-romu]

After some hours, buffer fills and average value gets closer to the expected value. But if I restart Home Assistant again, it empties the buffer and returns unexpected values again.

These are the recorded values from the referenced temperature sensor (there is no evidence of restarting points):

[ThomDietrich]

Hey @marc-romu, I will have time to look at this over the weekend.

I didn't change anything from my configuration before nor after upgrading.

And just to be clear: You are certain that you didn't see this behavior prior to the upgrade?

First thoughts, this might be related to changes in the recorder, as there were no significant changes to the statistics component since 2021.12: https://github.com/home-assistant/core/commits/dev/homeassistant/components/statistics

[marc-romu]

And just to be clear: You are certain that you didn't see this behavior prior to the upgrade?

Sure!

[ThomDietrich]

Does leaving out "scan_interval: 1800" make a difference?

[marc-romu]

No. I restarted with this configuration and the behaviour is the same:

- platform: statistics
  name: Temperatura 
  entity_id: sensor.temperatura
  state_characteristic: average_linear
  sampling_size: 10000
  precision: 1
  # scan_interval: 1800
  max_age:
    days: 1

[KonTiki1957]

hi to everyone. not sure if my problem with the energy dashboard is related to this issue but I report it anyway because it might be:

The values I feed to the dashboard are coming via MQTT from a PZEM (inflow) and from the inverter (SMA old model) which sends the instant solar production values via Bluetooth, they are then translated in MQTT messages which are then sent to Homeassistant. The remaining values are extrapolated through algebraic formulas (templates, integrations utility meters and so on) which may render correct values or not but the main thing is that I had it running, while since the upgrade it doesn’t show anything any more (There is no data to show/Grid neutrality could not be calculated etc) while the numeric value (the calculated ones) are still correctly shown on the main page.

[marc-romu]

Hi @KonTiki1957! Do you use the statistics integration in your calculations?

By the way @ThomDietrich, I updated to 2022.2.2 (which has changes in the recorder integration), but the behaviour is the same. I also noticed that when I reload statistics entities without restarting, the problem doesn't occur.

[KonTiki1957]

AS a matter of fact I don't my architecture is as follows:

a sensor (sensor integration) through the template platform feeds a template integration sensor which finally feeds the HA energy interface. But I suspect my issue shares the same root.

On 2/5/22 10:00, Marc Roca wrote:

Hi @KonTiki1957 https://github.com/KonTiki1957! Do you use the statistics integration in your calculations?

By the way @ThomDietrich https://github.com/ThomDietrich, I updated to 2022.2.2 (which has changes in the recorder integration), but the behaviour is the same. I also noticed that when I reload statistics entities without restarting, the problem doesn't occur.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/65499#issuecomment-1030583228, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOHQXXJF5HFBAZOBEJSHMFDUZTRLHANCNFSM5NOMX6XQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[marc-romu]

I've found one relevant error in logs. It takes place just after restarting Home Assistant:

2022-02-07 08:15:25 ERROR (MainThread) [homeassistant.components.statistics.sensor] sensor.temperatura: parsing error, expected number and received None

[marc-romu]

I realized that some other statistics sensors are working OK in my instance... The problem only occurs with statistics based on "min_max" sensors (https://www.home-assistant.io/integrations/min_max/). I suppose that these min_max sensors get a NaN state during startup that breaks the statistics. I'll try to find out more, but I don't know how because it isn't recorded into database...

[ThomDietrich]

@marc-romu the statistics integration also offers min and max values: https://www.home-assistant.io/integrations/statistics/

[marc-romu]

@marc-romu the statistics integration also offers min and max values: https://www.home-assistant.io/integrations/statistics/

I know, but they do it in a different way. Statistics returns the max/min of historical values of one sensor, while min_max returns the max/min between the last state of two or more sensors.

Min_max can be replaced by a template sensor, so I'll try it and I'll tell you.

Thanks!

[marc-romu]

Confirmed! It is an issue with min_max platform. I changed this:

  - platform: min_max
    name: Temperatura
    type: mean
    round_digits: 1
    entity_ids:
      - sensor.temperature_1
      - sensor.temperature_2

By this:

  - platform: template
    sensors:
      temperatura:
        friendly_name: "Temperatura"
        unit_of_measurement: "°C"
        value_template: "{{ (
                              ( states('sensor.temperature_1')|float(default='0') +
                              states('sensor.temperature_2')|float(default='0')
                              ) / 2
                            )|float(default='0') }}"
        availability_template: "{{ not (
                                    is_state('sensor.temperature_1','unavailable') or
                                    is_state('sensor.temperature_1','unknown') or
                                    is_state('sensor.temperature_2','unavailable') or
                                    is_state('sensor.temperature_2','unknown')
                                    ) }}"

And now statistics sensor is not breaking at restart.