history_stats doesn't always reset

JonLaliberte commented 4 months ago

The problem

I have some history stat's sensors set up, like this:

  - platform: history_stats
    name: Pool SWG on today
    entity_id: switch.pool_swg_switch
    unique_id: 'pool_swg_on_today_history_stats'
    state: 'on'
    type: time
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"

Most of the time, this works great. But sometimes it gets stuck:

Here you can see that the sensor was at 125 from about 12pm to about 10am. At that time I restarted Home Assistant and it reset the sensor's value to 0, then an automation kicked in which started it accumulating time again.

The problem is, this automation depends on this sensor to know whether or not to run - so if I had missed this, the automation would have never turned the switch on.

I have seen this issue happen sporadically for maybe 6 months or so now. To be perfectly clear, I don't know that I would have noticed it much before that as I didn't use this sensor type as much before.

What version of Home Assistant Core has the issue?

core-2024.7.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

history_stats

Link to integration documentation on our website

https://www.home-assistant.io/integrations/history_stats/

Diagnostics information

No response

Example YAML snippet

- platform: history_stats
    name: Pool SWG on today
    entity_id: switch.pool_swg_switch
    unique_id: 'pool_swg_on_today_history_stats'
    state: 'on'
    type: time
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"



### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_

home-assistant[bot] commented 4 months ago

history_stats documentation history_stats source

bcutter commented 3 months ago

I discovered the same recently. Example: grafik

  - platform: history_stats
    name: masked
    entity_id: switch.masked
    state: "on"
    type: time
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"
    # Update 2022-10-25: "scan_interval" not working anyway
    scan_interval: 900

I roughly know when this behaviour started... grafik

...but not why.

It ultimately should "reset" at midnight, but it doesn't anymore.
It should update (accumulate) on a 30 seconds basis, but it doesn't. The incremental changes are way to huge (maybe lost in translation, but looking at the graph should make it clear: "rough steps upwards" instead of "smoothly increasing line").
It resets when HA Core gets restarted, like @JonLaliberte mentioned.

There was no change on 2024-08-04 according to my event log. Only:

Last update was HA Core from 2024.1.6 to 2024.2.5 on 2024-08-01.
Last HA Core restart was on 2024-08-03 4:22 p.m.
Assumption/rough summary: since the first HA Core restart after installing HA Core 2024.2 this issue occured for me.

Idea number one: As @JonLaliberte mentioned

I have seen this issue happen sporadically for maybe 6 months or so now.

that might match the HA 2024.2 release assuming @JonLaliberte installed it right in Februar - just a wild guess.

Idea number two: We both use the same start: "{{ now().replace(hour=0, minute=0, second=0) }}". Maybe something in HA Core's internals has changed when operating on these?

Anyway, a dev needs to have a look at this as I (we) can only provide assumptions and indicators or wild guesses. For me currently all my history_stats sensors are useless as not reliable

issue-triage-workflows[bot] commented 2 weeks ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

bcutter commented 2 weeks ago

@JonLaliberte are you still seeing this issue on your side?

JonLaliberte commented 1 week ago

@bcutter The sensor I noticed this most with is not in use now (it's for the pool, and the pool is closed for the summer). I last noticed it on September 17 when I commented over here with an annotated screenshot: https://github.com/home-assistant/core/issues/114762#issuecomment-2356345198

I see there is a draft PR that is meant to resolve the issue, but I can't confirm that it is valid, and I'm not 100% sure these are caused by the same problem (though it seems likely).

karwosts commented 1 week ago

@JonLaliberte - could you estimate about what percentage of the time the sensor fails overnight?

JonLaliberte commented 1 week ago

@karwosts I'd say it was happening between 0-2 times per month. In the example I mentioned it happened 2 days in a row.

I checked another history stats sensor that is in active use (though doesn't have data every day as the switch isn't used daily) and it failed on the same days as this one did:

history stats did not reset history graph

I have not seen this happen since September 16th though. So maybe it was fixed somewhere... I'm going to set up an automation to notify me if this problem is detected going forward.

karwosts commented 1 week ago

I've setup a similar entity on my dev instance I'll monitor for a little while, with some extra debug logging, see if I can reproduce this at all. (unfortunately doesn't have much logging by default so there's not much to check when it doesn't work).

Must be some subtle timing related issue but looking at the code I couldn't obviously see anything.

karwosts commented 6 days ago

Also trying another sensor that "resets" every 10 minutes, since that might be more likely to trigger than something once a day. Haven't seen an issue so far though. This paired with a sensor that alternates 3 minutes on / 7 minutes off.

start: >
{% set start_minute = ((now().minute/10) | int) * 10 %}
{{ now().replace(minute=start_minute|int, second=0) }}

end: >
{{ now() }}

bcutter commented 6 days ago

Note: unchanged for me. But I'm not on latest HA Core release yet.

Edit: I also noticed sensors don't increase (interval is set to 900 so 15 minutes). Input sensor has been in watched state for more than one hour, the history stats sensor still reports nothing after more than one our. Is some kind of heavy delay invoked here? Super strange.

home-assistant / core