Ignore state change events while Home Assistant is starting and initializing entity states, override controllers correctly on startup

ndbroadbent commented 1 year ago

Ignore state change events for 70 seconds after starting
Check overrides 65 seconds after starting (when entity states should be ready)

Not sure about the hardcoded STARTUP_DELAY = 70, since every installation will have different startup times and entities from different integrations. This is probably quite specific to my HA instance running on a server, and with most of my entities coming from Zigbee2MQTT. I also wouldn't want to need to configure this for every single entity controller instance. Not sure if there's a nice way to make this configurable.

Description

This fixes an annoying issue I've experienced where I can't work on my HA config or perform upgrades late at night, because it would always turn on the bedroom lights while my wife was sleeping.

Checklist

[X] The PR title is clear, concise and follows conventional commit formatting.
[X] Double-check your branch is based on develop and targets develop
[X] Issue raised to compliment this PR (if no pre-existing issue exists)
[X] Code is commented, particularly in hard-to-understand areas and relevant issues are referenced.
[ ] Documentation repository updated to reflect new features or changes in behaviour (VERY IMPORTANT, undocumented features cannot be discovered and used!)
[X] Description explains the issue/use-case resolved and auto-closes related issues.
[ ] Breaking changes or changes in behaviour are called out and discussed in separate issues.
[ ] Testing of new changes completed by person who raised the issue.

Please open an issue before embarking on any significant pull request, especially those that add a new library or change existing tests, otherwise you risk spending a lot of time working on something that might not end up being merged into the project.

License

By submitting a patch, you agree to allow the project owners to license your work under the terms of the project license. Thank you for contributing!

Related Issues

Closes #285 Also: #231

ndbroadbent commented 1 year ago

I updated the PR with a new commit that adds a new pending state which is used while waiting for the startup delay. This looks a lot nicer in the UI, and it also helps to prevent any race conditions or edge cases. I think this approach is a bit better than my first attempt.

Pending state while HA is starting

After STARTUP_DELAY, all controllers instantly switch to the correct state:

danobot commented 1 year ago

Wow this is amazing, thanks for contributing. I will test on my instance for a bit before merging.

danobot commented 1 year ago

My main concern is whether this will make developing EC more dififcult because a restart takes 70 seconds. Option 1: ~I'm pretty sure Home Assistant allows you to register a call back on the entity to be called after the entity is fully registered and loaded in to the Home assistant entity registry. You can run your code within that function. It's better than relying on a fixed timer.~ I just realised that this irrelevant because we actually need to know about the other dependent entities, not our own EC entity.

Option 2: would be to add all input entities from configuration to a list and periodically check those entities (e.g. every 15 seconds). As long as the entities are not ready, EC can be in pending state. Once all entities have a real sensor value, EC can go to the correct state. Is there any harm is calling the startup_delay_callback function multiple times periodically?

ndbroadbent commented 1 year ago

Thanks for the feedback @danobot! I like the idea of adding all entities to a list and checking them periodically. I'm not familiar with the entity states while HA is starting up, but can check to see if they are all in the 'unavailable' state or something like that.

On a related note, I've been struggling to get the test suite running on my machine. I'd like to add tests for any changes and make sure I'm not breaking anything. But I'm getting an error when I run the command in dev.md:

#8 39.82 Failed to build pygraphviz
#8 40.27 Installing collected packages: urllib3, idna, chardet, certifi, typing-extensions, six, requests, pycparser, oauthlib, multidict, zipp, yarl, requests-oauthlib, MarkupSafe, isodate, idna-ssl, cffi, azure-core, attrs, async-timeout, typing, sgmllib3k, pytz, python-engineio, pyparsing, pycares, ordered-set, msrest, Jinja2, importlib-metadata, dataclasses, cryptography, azure-mgmt-core, azure-common, aiohttp, websocket-client, voluptuous, uvloop, tomli, sockjs, pyyaml, python-socketio, python-dateutil, pygments, py, pluggy, pid, paho-mqtt, packaging, iso8601, iniconfig, feedparser, deepdiff, cchardet, bcrypt, azure-storage-blob, azure-mgmt-storage, azure-mgmt-resource, azure-mgmt-compute, azure-keyvault-secrets, astral, aiohttp-jinja2, aiodns, watchdog, transitions, pytest, pygraphviz, mock, docopt, colorama, appdaemon, pytest-watch, pytest-mock, freezegun, appdaemontestframework
#8 51.99     Running setup.py install for pygraphviz: started
#8 52.44     Running setup.py install for pygraphviz: finished with status 'error'
#8 52.45     ERROR: Command errored out with exit status 1:
#8 52.45      command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qnbfedit/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.6m/pygraphviz
#8 52.45          cwd: /tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/
#8 52.45     Complete output (34 lines):
#8 52.45     /usr/local/lib/python3.6/site-packages/setuptools/dist.py:700: UserWarning: Usage of dash-separated 'build-requires' will not be supported in future versions. Please use the underscore name 'build_requires' instead
#8 52.45       % (opt, underscore_opt))
#8 52.45     running install
#8 52.45     Trying dpkg
#8 52.45     Trying pkg-config
#8 52.45     Package libcgraph was not found in the pkg-config search path.
#8 52.45     Perhaps you should add the directory containing `libcgraph.pc'
#8 52.45     to the PKG_CONFIG_PATH environment variable
#8 52.45     No package 'libcgraph' found
#8 52.45     Traceback (most recent call last):
#8 52.45       File "<string>", line 1, in <module>
#8 52.45       File "/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup.py", line 90, in <module>
#8 52.45         tests_require=['nose>=1.3.7', 'doctest-ignore-unicode>=0.1.2', 'mock>=2.0.0'],
#8 52.45       File "/usr/local/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
#8 52.45         return distutils.core.setup(**attrs)
#8 52.45       File "/usr/local/lib/python3.6/distutils/core.py", line 148, in setup
#8 52.45         dist.run_commands()
#8 52.45       File "/usr/local/lib/python3.6/distutils/dist.py", line 955, in run_commands
#8 52.45         self.run_command(cmd)
#8 52.45       File "/usr/local/lib/python3.6/distutils/dist.py", line 974, in run_command
#8 52.45         cmd_obj.run()
#8 52.45       File "/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup_commands.py", line 43, in modified_run
#8 52.45         self.include_path, self.library_path = get_graphviz_dirs()
#8 52.45       File "/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup_extra.py", line 158, in get_graphviz_dirs
#8 52.45         include_dirs, library_dirs = _try_configure(include_dirs, library_dirs, _pkg_config)
#8 52.45       File "/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup_extra.py", line 113, in _try_configure
#8 52.45         i, l = try_function()
#8 52.45       File "/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup_extra.py", line 65, in _pkg_config
#8 52.45         output = S.check_output(['pkg-config', '--libs-only-L', 'libcgraph'])
#8 52.45       File "/usr/local/lib/python3.6/subprocess.py", line 356, in check_output
#8 52.45         **kwargs).stdout
#8 52.45       File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
#8 52.45         output=stdout, stderr=stderr)
#8 52.45     subprocess.CalledProcessError: Command '['pkg-config', '--libs-only-L', 'libcgraph']' returned non-zero exit status 1.
#8 52.45     ----------------------------------------
#8 52.45 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-6z9wa0qp/pygraphviz_dc1da0ffa64f44cb9feffcdb55cf97b7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qnbfedit/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.6m/pygraphviz Check the logs for full command output.
#8 52.80 WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
#8 52.80 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/sh -c pip install -r /tmp/requirements.txt]: exit code: 1

danobot commented 1 year ago

I never got the unit tests to work. Back when I looked into it there were some challenges mocking the "passage of time" to validate EC does a thing after timeout. See issue #101. This was a huge blocker because everything depends on timers and timeouts in EC and I eventually gave up on unit tests altogether. My process now is to run any change in my prod HA instance and observe for a couple of days

danobot commented 1 year ago

@ndbroadbent is this still good to merge? I ran this version on my instance for months and forgot about it

andylokandy commented 1 year ago

Also waiting on this, it really helps!

ndbroadbent commented 1 year ago

@danobot I am still running this branch on my instance as well and it hasn't given me any problems so far. I think this approach is a bit unsophisticated and could be refined, as you mentioned:

Option 2: would be to add all input entities from configuration to a list and periodically check those entities (e.g. every 15 seconds). As long as the entities are not ready, EC can be in pending state. Once all entities have a real sensor value, EC can go to the correct state. Is there any harm is calling the startup_delay_callback function multiple times periodically?

I like that idea better if it's possible, but not sure how to implement it. I guess we can check for when a sensor changes from unavailable to on/off? Maybe could also have a timeout in case the sensor stays unavailable and show an error.

I would also really like to help with unit tests, I've been wanting to contribute a few more features but it would be really nice to have some test coverage. I am a Ruby on Rails developer during the day, and we have a library called timecop that is used for mocking and stubbing time. I saw there's a pytimecop port that might be helpful. This way you can control the time precisely in the tests and don't need to wait for the actual amount of time. So that might be a good way to go

danobot commented 1 year ago

Mate I tried to get tests going before and eventually gave up due to time limitation. if you could establish the test pattern using pytimecop that would be great. Given that there is currently no better solution for your issue and we haven't experienced any problems I will merge this PR.

danobot / entity-controller