TOD incorrect after midnight restart

toogooda commented 2 years ago

The problem

binary sensor TOD set to off when should be on after 2am restart

 - platform: tod
    name: lateNightTime
    after: '21:30'
    before: sunrise

What version of Home Assistant Core has the issue?

2022.2.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

TOD

Link to integration documentation on our website

here

Diagnostics information

No response

Example YAML snippet

- platform: tod
    name: lateNightTime
    after: '21:30'
    before: sunrise

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

tod documentation tod source _{^{(message by IssueLinks)}}

bleem313 commented 2 years ago

I am having the exact same problem, also with a daily reset at 2 am. Once the system comes back online TOD changes the sensor to Off using the Night sensor example found in the documentation.

binary_sensor:
  - platform: tod
    name: Night
    after: sunset
    before: sunrise

As you can see in the sensor history, it turns off just after 2 am which is when HA restarts.

I am currently on Home Assistant OS 7.5 with core-2022.3.7 and supervisor-2022.03.5

DarkWarden85 commented 2 years ago

Hi, I also have the same problem. The HA restart was caused by a power outage at around 1h42. Normally, the tod sensor should have stayed on till around 4h48. Screenshot_20220620-052411

DmytroKorniienko commented 2 years ago

All tod sensors going to off after core restart.

Screenshot_2022-09-05-04-53-35-748_com android chrome Screenshot_2022-09-05-04-54-08-032_com android chrome Screenshot_2022-09-05-04-54-24-814_com android chrome Screenshot_2022-09-05-04-56-28-753_com android chrome

ahaverty commented 2 years ago

I had similar issues last night/today after some restarts beyond midnight.

binary_sensor:
  - platform: tod
    name: Night
    after: sunset
    before: sunrise

ahaverty commented 2 years ago

I hit this again last night. After a restart of HA at around midnight, my

    after: sunset
    before: sunrise

sensor returned to off incorrectly since.

It's fairly impactful, as I have a good few of my lights automations working off TOD conditions, and seemingly have no way to manually fix the values on these sensors, since they're defined in yaml.

I can see older, closed, but related issues at #50315 and #24577

ahaverty commented 2 years ago

Does anyone have a workaround or alternative integration that follows the sun? I'm hitting this regularly, as I'm doing a good bit of late night tweaking recently.

DmytroKorniienko commented 1 year ago

@ahaverty

Does anyone have a workaround or alternative integration that follows the sun? I'm hitting this regularly, as I'm doing a good bit of late night tweaking recently.

  - binary_sensor:
    # tod workaround
    - name: "Night"
      unique_id: tod_night
      state: >
        {% set cur_sunset = iif(state_attr("sun.sun", "next_setting")>state_attr("sun.sun", "next_rising")
        ,as_datetime(state_attr("sun.sun", "next_setting"))-timedelta(days=1)
        ,as_datetime(state_attr("sun.sun", "next_setting")))%}

        {% if 
        now()>cur_sunset
        and now()<as_datetime(state_attr("sun.sun", "next_rising"))
        -%}
          on
        {%- else -%}
          off
        {%- endif %}

DarkWarden85 commented 1 year ago

I had an unexpected restart at 00:38 AM in the morning. All my tod sensors lost their state. Here's a picture of a tod sensor controlling my blinds. Screenshot_20221214-090429 Notice how it turns of at 00:38 in the morning without any corresponding logbook entry. I have another tod sensor which does not depend on the sun but uses fixed hours which does not have that problem. Can anybody else confirm that the problem only occurs when a tod sensor depends on the sun?

ahaverty commented 1 year ago

@DmytroKorniienko thanks for sharing! I'm stuck though with some more complex TOD's that are important to me for outdoor light automations. I had a go at modifying your code for evening for example, but couldn't find a way to get the after_offset working properly.

binary_sensor:
  - platform: tod
    name: Night
    after: sunset
    before: sunrise

  - platform: tod
    name: Day
    after: sunrise
    before: sunset

  - platform: tod
    name: Evening
    after: sunset
    after_offset: "-00:30"
    before: "00:30"

  - platform: tod
    name: Outdoor Evening
    after: sunset
    before: "23:30"

  - platform: tod
    name: Early Morning
    after: sunrise
    before: sunrise
    before_offset: "02:00"

What I tried:

{% set cur_sunset = iif(state_attr("sun.sun", "next_setting")>state_attr("sun.sun", "next_rising")
        ,as_datetime(state_attr("sun.sun", "next_setting"))-timedelta(days=1)
        ,as_datetime(state_attr("sun.sun", "next_setting")))%}

        {% if now()>(cur_sunset - timedelta(minutes = 30)) and now()<today_at("00:30") -%} on {%- else -%} off {%- endif %}

Side note, is this a complex bug for HA to solve properly, or is it just low priority/forgotten? Maybe someone knows who to reach out to for a fix?

DarkWarden85 commented 1 year ago

The problem is still present with version 2022.12.8. My home assistant instance suffers from regular unplanned restarts during the night, the cause of which I haven't found yet. This would not really be an issue if it weren't for all my time of day sensors having a wrong state after the restart. Is there really no fix for this?

DmytroKorniienko commented 1 year ago

@DmytroKorniienko thanks for sharing! I'm stuck though with some more complex TOD's that are important to me for outdoor light automations. I had a go at modifying your code for evening for example, but couldn't find a way to get the after_offset working properly.

after_offset requires more complex logic and I'm not implemented it for my automations, but can share one more example:

  - binary_sensor:
    # from 21:00 to 5 minutes before next_rising
    - name: "Light Control"
      unique_id: tod_light_control
      state: >
        {% set cur_sunset = iif(state_attr("sun.sun", "next_setting")>state_attr("sun.sun", "next_rising")
        ,as_datetime(state_attr("sun.sun", "next_setting"))-timedelta(days=1)
        ,as_datetime(state_attr("sun.sun", "next_setting")))%}
        {% set cur_start = strptime(as_timestamp(cur_sunset) | timestamp_custom('%Y-%m-%d 00:00:00%z'),'%Y-%m-%d %H:%M:%S%z') %}

        {% if 
        now()>cur_start+timedelta(hours=21, minutes=00)
        and now()<as_datetime(state_attr("sun.sun", "next_rising"))-timedelta(hours=0, minutes=5)
        -%}
          on
        {%- else -%}
          off
        {%- endif %}

DarkWarden85 commented 1 year ago

Issue is still present on Home Assistant 2023.1.2. I had an unforeseen restart in the middle of the night and all my sunbased tod-sensors simply turned off. I included a pic of the tod-sensor handling my covers. All my covers stayed closed this morning instead of reacting to sunrise.

Screenshot_20230112-084750

ahaverty commented 1 year ago

@frenck sorry for the direct @, but perhaps nobody at HA is assigned/tracking TOD issues. I'd appreciate hearing if you think this issue will be eventually triaged by anyone at HA? Thanks!

frenck commented 1 year ago

but perhaps nobody at HA is assigned/tracking TOD issues

@ahaverty There is no code owner for this integration, so that is correct.

I'd appreciate hearing if you think this issue will be eventually triaged by anyone at HA?

Home Assistant is an open source project and relies on contributions. Feel free to jump in an contribute a fix.

../Frenck

ahaverty commented 1 year ago

Thanks @frenck 👍

For anyone else trying to fix this: I took a stab at it, but unfortunately, I can't even reproduce the issue in unit tests.

As far as I understood, my test case here should fail after the HA restart, based on what I've experienced with my actual HA instance restarting just after midnight: https://github.com/ahaverty/core/commit/ecb5353025a9bc14cac41594796406801db9b362

I did see there was a recent cleanup of TOD in #79412, so I tested against an older version of HA (an Oct 2022 release) just in case that PR has already fixed this issue and we haven't noticed since, but I still couldn't get my test to fail on an older Oct release.

My theory on when the issue occurs (and what I was trying to fail in unit tests):

When a sun-based TOD sensor starts on the day previous to a restart, but ends after the restart I've found that sun-based TOD sensors work fine for restarts during the day, for configurations like 'daytime', so my guess is there's something not being picked up during initialisation of the sensors after a restart. I'm just unfortunately not knowledgeable enough on HA to pinpoint/reproduce it here in code.

My theories on why I can't get the test to fail:

I'm misunderstanding what hass.async_stop & hass.async_start actually does
- Perhaps the async stop/start isn't a realistic replica of a real HA reboot.
This restart issue is more complex to reproduce than I expected (I did also include offsets in my test, just in case that's required for this issue, but the test still passes with them)

EDIT: No, I have an obvious issue with a couple of things with my test case above. It looks like the suite is setup for sunsets happening before midnight for some reason. ~I'm now able to fail the test when I change the timezone and location to Ireland, against the Oct release. I'll look at testing against dev branch next, and see if I can add a fix also (i.e the hard part 😬)~

EDIT 2: yes, there's something odd going on with this test suite if I'm reading this correctly. The timezone is set to the US, but the lat/lng is in Poland. So the sunrise/sunsets aren't going to be realistic at all 🤔

EDIT 3: I'm still unable to reproduce this in unit tests unfortunately. As far as I can tell, I'm calling the correct restart methods. I've also fixed the suite's lat/lng to match the US Pacific timezone, so that sunrise/sunset happen on different days (as typically expected for most of the world). If anyone else wants to try fix, perhaps have a look at my attempt first: https://github.com/home-assistant/core/compare/dev...ahaverty:core:fix/TOD-incorrect-after-midnight-restart

EDIT 4: Nope, I've gone back to an Oct '22 release again with my latest test fixes, and can't reproduce the issue in unit tests. I've also tried:

Testing with my exact TOD configuration, with multiple TOD sensors (below).
Starting the hass async_stop before midnight, jumping to after midnight, and starting hass again

I'm stumped..

My TOD config for reference:

config = {
        "binary_sensor": [
            {
                "platform": "tod",
                "name": "Night",
                "after": "sunset",
                "before": "sunrise",
            },
            {
                "platform": "tod",
                "name": "Day",
                "after": "sunrise",
                "before": "sunset",
            },
            {
                "platform": "tod",
                "name": "Evening",
                "after": "sunset",
                "after_offset": "-00:30",
                "before": "00:30",
            },
            {
                "platform": "tod",
                "name": "Outdoor Evening",
                "after": "sunset",
                "before": "23:30",
            },
            {
                "platform": "tod",
                "name": "Day",
                "after": "sunrise",
                "before": "sunset",
            },
            {
                "platform": "tod",
                "name": "Early Morning",
                "after": "sunrise",
                "before": "sunrise",
                "before_offset": "02:00"
            }
        ]
    }

DarkWarden85 commented 1 year ago

Hi @ahaverty,

Thank you for going through so much trouble to help get this fixed. It is odd that the problem doesn't occur during unit tests. I have currently three sun-based tod-sensors, which fail reliably after a midnight restart (last one was at 00:38). Maybe you could try with that config?

binary_sensor:
  - platform: tod
    name: Dachfenster Nachtmodus
    after: sunset
    before: sunrise
  - platform: tod
    name: Jalousien Nachtmodus
    after: sunset
    after_offset: "00:35:00"
    before: sunrise
    before_offset: "-00:35:00"
  - platform: tod
    name: Jalousien Südseite RDC Nachtmodus
    after: sunset
    before: sunrise
    before_offset: "-00:10:00"

Time Zone: (GMT+01:00) Luxembourg

Elevation: 420 meters

Country: Luxembourg

Coordinates: Somewhere in Luxembourg (I'm hesitant to post the coordinates of my home here in public)

ahaverty commented 1 year ago

Thanks @DarkWarden85, we practically have the same tod sensor there, which I've covered already in testing. I'm also having the restart issue myself in production consistently, so I don't think it's my config that's the problem for this test 🤔 Thanks though!

ahaverty commented 1 year ago

Although @DarkWarden85 just re-reading back through people's comments here. I'm assuming based on languages a bit, but the majority of us seem to be located around Europe. I have found it odd that this issue isn't more popular/fixed, maybe it is something unusual like:

TOD component initialising with a default US timezone on boot, before HA had set the correct timezone (hence why it thinks the sensor is off) 🤷‍♂️
- I'm not sure that's easy to prove in unit testing either, since the tests are overriding HA's timezone settings by default

Maybe if anyone here is based in the US, and also has this post-midnight issue, they could comment to let us know👍

DarkWarden85 commented 1 year ago

Looking at your unit-tests everything seems to be in-line with the other unit-tests. Maybe there is something odd with the force stop and restart where it initializes itself with a different time when restarting. Maybe you should set the time again after the restart. This is just a wild guess and I'm not that great at programming in the first place

Edit: Stupid suggestion from me with setting the time again after a restart. This should have no effect because the init during restart is what causes the problem. But still... is it using the correct testing time after the restart?

wafliron commented 1 year ago

@ahaverty FWIW I experience this issue in Australia too - reliably reproducible with an after-midnight HA reboot. Another data point towards it potentially being something to do with non-US timezones - or perhaps non-US locales/languages?

Random thought, but possible it's got something to do with the environment HA is running in too? I'm using Home Assistant Operating System running as an ESXi virtual machine (think I downloaded an OVF from somewhere and used that, but so long ago I don't remember properly).

DarkWarden85 commented 1 year ago

@wafliron to add to this, I'm running Home Assistant OS on a raspberry pi 4 while also experiencing the issue.

ahaverty commented 1 year ago

I may as well add to the list too in case anyone else beats me to fixing:

Ireland
Home assistant Blue (supervised, usually running close to the latest minor patches)

ahaverty commented 1 year ago

I tried the unit tests with my timezone and lat/lng in Ireland, but unfortunately couldn't get the restart to fail the test.

ahaverty commented 1 year ago

Hmm, I have a (possibly messy) workaround idea that might be worth spitballing with ye (while we wait for a saviour to fix the issue).

Since this seems to only affect sun-based TOD's that end one day later than the start (right?)

I wonder if I split them into two parts, for a night TOD for example:

- platform: tod
  name: night_a
  after: sunset
  before: "23:59:59"

- platform: tod
  name: night_b
  after: "00:00"
  before: sunrise

And then created a group for these two called "Night" via a group UI in helper settings.

Edit: Hmm, it'll probably cause havoc for that 1 second gap around midnight, for anyone with automations listening to state changes for light automations etc.

bleem313 commented 1 year ago

I have more information listed in the 3rd comment on this but I will include my config again here. I'm US-based with a US timezone currently running on a raspberry pi 4 and do indeed have this issue, so I don't think being non-US is a cause. I had this issue on a pi 3 as well.

binary_sensor:
  - platform: tod
    name: Night
    after: sunset
    before: sunrise

DarkWarden85 commented 1 year ago

I just wanted to confirm that the bug is still present in the newest version 2023.2.0, so no need for @ahaverty to downgrade your testing version.

Here's the result from a scheduled restart at 00h30 AM on the prior version 2023.1.7:

Screenshot_20230202-063645

The same happens with a scheduled restart at 00h30 AM on version 2023.2.0 :

Screenshot_20230203-063529

In both instances, the sensor was only turned on until 00h30 AM in the morning and it turned off after the restart

zzachattack2 commented 1 year ago

but perhaps nobody at HA is assigned/tracking TOD issues

@ahaverty There is no code owner for this integration, so that is correct.

I'd appreciate hearing if you think this issue will be eventually triaged by anyone at HA?

Home Assistant is an open source project and relies on contributions. Feel free to jump in an contribute a fix.

../Frenck

As this integration has been “broken” for over a year, and apparently with no code owner to remedy, why does it even remain part of the core? Seems like the integration should either function, or progress to functioning, or just not exist at all…

ahaverty commented 1 year ago

Very untested, but I think I've come up with a decent workaround for the likes of a "night" sensor that relies on sunrise and sunset:

binary_sensor:
  - platform: tod
    name: left_night
    after: sunset
    before: "23:59"

  - platform: tod
    name: right_night
    after: "00:00"
    before: sunrise
    before_offset: "-00:02"

template:
    - binary_sensor:
      - name: "Night"
        unique_id: night
        delay_off:
          minutes: 2
        state: >
          {{ (is_state('binary_sensor.left_night', 'on') or is_state('binary_sensor.right_night', 'on')) }}

_i.e: The left_night and right_night I don't intend to use anywhere other than in the template's state for binarysensor.night above.

I've tested restarting just after midnight, and see binary_sensor.right_night is set to 'on' as expected. I've gone with 23:59 to be safe, but added a delay_off of 2 minutes so that the night sensor doesn't actually change to 'off' too early or trigger any automations just before midnight.

And the -2min before_offset is to counteract that delay_off. So it should function exactly like the standard night sensor, but without the nightime restart issues.

Edit July 2024: I've been using this without any issues since 2023 in case anyone is wondering if they should use it 👍

DarkWarden85 commented 1 year ago

@zzachattack2 I agree that the non-functioning part of tod should be removed, but how to do that with no code maintainer for it? Also, there is the problem that tod might break completely in the future if there are any changes to core functionality it relies on.

@ahaverty Thanks for that workaround, it's the easiest to setup I have seen so far. Will try it out too.

DarkWarden85 commented 1 year ago

@ahaverty I tried your workaround this night. It also works on my HA instance :D. Here's the comparison: Screenshot_20230315-071634

I made a restart at 00:30 in the morning. You can see that the workaround sensor stays on after the restart and the original one doesn't.

However I did a slight modification to your approach. I did not specify a delay_off of 2 minutes in the template sensor. Instead I have the right tod sensor trigger at 00:00 with an offset of -00:02:00. I think that makes it a bit easier. Here's my config:

binary_sensor:
  - platform: tod
    name: Jalousien Nachtmodus Rechts
    after: sunset
    after_offset: "00:35:00"
    before: "23:59"
  - platform: tod
    name: Jalousien Nachtmodus Links
    after: "00:00"
    after_offset: "-00:02"
    before: sunrise
    before_offset: "-00:35:00"

template:
  - binary_sensor:
      - name: "Jalousien Nachtmodus workaround"
        unique_id: template_binary_Sensor_jalousien_nachtmodus_workaround_unique_id
        state: >
          {{ (is_state('binary_sensor.jalousien_nachtmodus_links', 'on') or is_state('binary_sensor.jalousien_nachtmodus_rechts', 'on')) }}

PS: I named my sensors wrong and confounded left with right, but it still works, lol.

igd001 commented 1 year ago

I have the same situation with several night sensors created with TOD. I am based in Spain. HA version 2023.3.6 running on Yellow.

My workaround has been to create a complementary day sensor in my binary_sensor section

- platform: tod
  name: day_30
  after: sunrise
  after_offset: "+00:30"
  before: sunset
  before_offset: "-00:30"

and calculate the corresponding night sensor as a template binary_sensor

- name: "night_30"
  state: "{{ is_state('binary_sensor.day_30', 'off') }}"

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

DarkWarden85 commented 1 year ago

This issue still persists. Please do not autoclose.

amzaldua commented 1 year ago

Hi, I have the same issue. I have HAOS version 2023.9.2 running over an Intel Nuc and living in Canary Islands (Spain) Is my geographical situation relevant? The problem is that the status of the TOD binary sensor is not retain after restart.

This is my binary_sensor code:

binary_sensor:
  - platform: tod
    name: Trevina Noche
    before: sunrise
    before_offset: "-00:10:00"
    after: sunset
    after_offset: "00:10:00"

amzaldua commented 1 year ago

I conducted some tests, and confirm the issue arises when the binary sensor's state was set to 'on' the previous day. Performing a restart on the same day does not alter the state, but doing so the following day resets it to its default state, 'off'. My solution was to define a Day sensor instead Night sensor:

binary_sensor:
  - platform: tod
    name: Trevina Dia
    before: sunset
    before_offset: "00:10:00"
    after: sunrise
    after_offset: "-00:10:00"

DarkWarden85 commented 10 months ago

@amzaldua I can confirm that defining a tod sensor whose ON-state doesn't span from one day to the next doesn't have any issues. Thanks for finding this workaround.

Sithembisogama commented 8 months ago

Upon restart the tod platform has no memory of when the previous update (should have) happened, it only looks forward. As a workaround I have written this template which resolves true when time is between sunset and sunrise. The idea is to use it as an automation condition when home assistant starts, then update the sensor programmatically:

sensor

- platform: tod
  name: Sunset-to-sunrise
  after: sunset
  after_offset: "00:00:00"  #starting time for "On" state
  before: sunrise
  before_offset: "00:00:00" #finish time for "On" state

template

{# check if time right now is between sunset and sunrise #}
{% set timenow=now() %}
{% set sensor_name = 'binary_sensor.sunset_to_sunrise' %}
{% set nextupdate = state_attr(sensor_name, 'next_update') %}
{% set before_ = state_attr(sensor_name, 'before') %}
{% set after_ = state_attr(sensor_name, 'after') %}
{{ nextupdate==before_ and timenow.timestamp()<as_timestamp(nextupdate) }}

issue-triage-workflows[bot] commented 5 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

DarkWarden85 commented 5 months ago

This bug is still present, so there is no reason to close it.

nopbop commented 2 months ago

Same issue for me

home-assistant / core