OpenTherm Gateway no longer working in 2022.7.0 (Python 3.10)

ReinkNL commented 2 years ago

The problem

Integration is not working, log is flooded with errors, probably due to the upgrade to Python 3.10

What version of Home Assistant Core has the issue?

2022.7.0

What was the last working version of Home Assistant Core?

2022.6.7

What type of installation are you running?

Home Assistant OS

Integration causing the issue

OpenTherm Gateway

Link to integration documentation on our website

https://www.home-assistant.io/integrations/opentherm_gw/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Error doing job: Exception in callback SerialTransport._read_ready()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/site-packages/serial_asyncio/__init__.py", line 120, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.10/site-packages/pyotgw/protocol.py", line 95, in data_received
    self._readbuf += data
AttributeError: 'protocol' object has no attribute '_readbuf'

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

opentherm_gw documentation opentherm_gw source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 2 years ago

Hey there @mvn23, mind taking a look at this issue as it has been labeled with an integration (opentherm_gw) you are listed as a code owner for? Thanks! _{^{(message by CodeOwnersMention)}}

andriej commented 2 years ago

Can confirm same issue here.

mvn23 commented 2 years ago

Thanks for the report. It does look like a dependency issue in pyotgw indeed. I am on vacation at the moment, so I can't really look into it until at least the end of next week.

For now all you can do is either downgrade or disable the integration.

acfnews commented 2 years ago

Can confirm same issue here.

Exact same issue here. Downgraded for the moment. Looking forward for a fix, and testing the fix.

Sandolution commented 2 years ago

Same here, luckily found this issue to know it was OpenTherm GW spitting out this error. Happy to test any upcoming fixes - luckily it's still warm outside.

frenck commented 2 years ago

🤚 I have the same issue! 👇 Read below

Hi there! Sorry to read you are experiencing the same issue as more people here. Nobody likes having issues 😞

Finding an issue on a GitHub issue tracker that matches your problem is kinda nice: At least you know you are not alone. So, let's leave a commit with: "Yeah, I have the same issue", or "+1!!!". 🚫 No!

Please do not create "I have the same issue" comments!

Not just this issue or this project, but anywhere on GitHub or any issue tracker on the internet even. It doesn't add to triaging the issue. It only generates noise when reading / triaging the case.

What to do instead?

Instead, go to the top of the issue, and add a 👍 emoji reaction. That way, we can still see that you (and how many others) experience this issue while keeping the issue threads nice and clean.

Thanks! ❤️

../Frenck

mvdnes commented 2 years ago

The first error occurring in OpenTherm Gateway is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/site-packages/pyotgw/protocol.py", line 44, in connection_made
    self._cmd_lock = asyncio.Lock(loop=self.loop)
  File "/usr/local/lib/python3.10/asyncio/locks.py", line 78, in __init__
    super().__init__(loop=loop)
  File "/usr/local/lib/python3.10/asyncio/mixins.py", line 17, in __init__
    raise TypeError(
TypeError: As of 3.10, the *loop* parameter was removed from Lock() since it is no longer necessary

This causes _readbuf being undefined later on.

The master branch of pyotgw has a fix for this issue, but unfortunately this has not yet been published in a release.

voyagertm commented 2 years ago

Hello @frenck

For me I use Wifi connection instead of usb. But I have the same issue, so not sure if that usb problem wil fix it too, I hope it wil.

andriej commented 2 years ago

@voyagertm it's the same issue

caveman85 commented 2 years ago

I have disable the OTGW intergration at this moment, but receive error:

  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 240, in _async_init
    flow = await self.async_create_flow(handler, context=context, data=data)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 776, in async_create_flow
    raise data_entry_flow.UnknownHandler from err
homeassistant.data_entry_flow.UnknownHandler

Maybe also a Python issue?

mvn23 commented 2 years ago

I have disable the OTGW intergration at this moment, but receive error:

  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 240, in _async_init
    flow = await self.async_create_flow(handler, context=context, data=data)
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 776, in async_create_flow
    raise data_entry_flow.UnknownHandler from err
homeassistant.data_entry_flow.UnknownHandler

Maybe also a Python issue?

This looks unrelated, especially since the integration was disabled. Is there any indication that it is related to the pyotgw dependency? If not, please open a separate issue.

sygys commented 1 year ago

@frenck I don't understand how these things can still happen. Every updates something essential breaks. You can blame the devs not updating to the latest py. But instead HA updates should keep all integrations intact and be backward compatible. You can't just release something and then say. To bad we will maybe fix it... Our whole home and climate is build around the otgw integration. It's the same as microsoft releasing an update that renders the whole office installment useless and then start bitching to people that complain. We depend on this software. becarefull what you break next time and when you update something big like a new python version. Be sure to have the old version in place for the integrations that do not yet work with them. This 8s been going on for a long time now. Pushing updates of supervisor which can't be shut down. Releasing updates of core that do more harm then good. Please split core open and let people update integrations and essential stuff like we can with add-ons so we can leave out things that don't work or revert back to an old integration. Now if we want the new feature of 1 integration we are forced to install all the ones that break too.

sygys commented 1 year ago

And another thing... If there are know issues 10 days ago about a core update it would be nice that when people press the update button a pop up will show warning them of the integrations they have installed that have know issues. And maybe even give them an option to subscribe to being notified when the integration is fixed

frenck commented 1 year ago

@frenck I don't understand how these things can still happen. Every updates something essential breaks

If no one tests it during beta, it will be hard to achieve that. Feel free to join or beta tests to help prevent these issues.

But instead HA updates should keep all integrations intact and be backward compatible

This is upstream breaking (Python in combination with the upstream package), not Home Assistant. You are mixing up things.

sygys commented 1 year ago

Just reverted back to corr 2022.6.7 and now I can't reach my file system anymore. I have this for a long time now. Everytime I revert back to an old core version the whole config/media etc folders aren't reachable anymore. I have to fully install ha again on my nvme and do a full restore... This is so anoying.

The whole config cant be reached everytime i revert back Knipsel

frenck commented 1 year ago

Everytime I revert back to an old core version the whole config/media etc folders aren't reachable anymore

Depending on the installation method you use. If you are using Home Assistant OS, please restore backups (instead of version reverts).

Which installation methods are you using?

johansmitsnl commented 1 year ago

@sygys I feel for the users that have breaking things but it is software and it does happen from time to time. If the experience you feel is like it breaks all the time. Ensure to have proper backups before you start updating. Then it is much simpler to revert back.

@frenck what I did find interesting is that (myself included) would like to see if a new update is available from HA it has confirmed issues that will prevent integrations to fail and is visible before you update. Then users can decide to wait for the update for it to be fixed in a update. This might give less :-1: feelings for some users.

This reply is not to have a solution for the issues but about the experience of the user which in the last releases from UI has improved so much (thanks the HA team and all it contributors :+1: :1st_place_medal: ) and this might be a good addition.

sygys commented 1 year ago

im using Hass OS. When i make a core backup when updating and then restore that core backup ithe config is gone. everytime. it has been for some time now. i wonder if im the only one.

frenck commented 1 year ago

@frenck what I did find interesting is that (myself included) would like to see if a new update is available from HA it has confirmed issues that will prevent integrations to fail and is known.

We link known and reported issues for each integration from the Home Assistant documentation AND from your own Home Assistant instance. This case was reported after Home Assistant 2022.7 was shipped. We offer backups for that reason, use them.

sygys commented 1 year ago

It seems like this is a rights thing. because with samba i can get into the config folders. thank god i can place an old total back up back and do a full restore. but i would really like to know if im the only one with the problem that they lose the rights when restoring a core backup

frenck commented 1 year ago

im using Hass OS. When i make a core backup when updating and then restore that core backup ithe config is gone. everytime. it has been for some time now. i wonder if im the only one.

Everytime I revert back to an old core version the whole config/media etc folders aren't reachable anymore. I have to fully install ha again on my nvme and do a full restore..

Backups should just work. Have you reported that issue?

Re-installation should never be needed. And honestly, I've re-installed my setup in the past four years just once! (By choice, not because it was needed). And I run nightly builds on my system even (meaning I upgrade my Home Assistant every day to a new release).

Please report backup issues so they can be addressed.

sygys commented 1 year ago

MAybe its more simple then i thought.

https://community.home-assistant.io/t/missing-config-directory-contents/414051

but god knows why home assistant stops showing files when an addon has protection mode off when restoring a core backup

Cant test it anymore though a full backup restore takes less then a minute with my samsung m2 980 pro :)

Still i would really like to see integrations that can be updates individually. I understand in this case it would have still been broken because of the python update. but it would be very nice to being able to update some integrations but leave others be. So just dont need an update if they work just fine. And second please make an option to opt-out supervisor updates. why are they even automatic? If the supervisor update is messed up and it updates and just at that moment you are on a holiday this could mess up allot

frenck commented 1 year ago

but god knows why home assistant stops showing files when an addon has protection mode off when restoring a core backup

That is absolutely not related.

Again, please report the backup issue in the Supervisor project.

Still i would really like to see integrations that can be updates individually.

That is not related to this issue, nor can we do that at this time. Also note, that that sounds ideal, but it isn't for many other reasons. Anyways, please use the community forums for feature requests.

And second please make an option to opt-out supervisor updates

Please use the community forums for that too, as that is a feature request not related to this issue.

farmio commented 1 year ago

This issue could have been easily avoided / detected by comprehensive unit tests. How much you mock away vs. how much of the libs code you use in the tests is mostly a balance decided by the codeowners. Maybe going a little bit deeper into the lib would have helped here.

On the other hand - if every integration would run their complete libs logics against HA CI this would probably increase build times a lot 😬

rob-on-git commented 1 year ago

@frenck @sygys : To be honest, i did had the same issue with disappearing config-directory. I upgrade to 2022.7.3 and discovered the issue with OpenTherm, so i tried to roll back a backup and screwed up the whole installation.

My luck was that i regularly take ZFS snapshots, so i could rely on that snapshot...

frenck commented 1 year ago

To be honest, i did had the same issue with disappearing config-directory.

That is unrelated to this issue. Please, report that in a separate issue so it can be looked at. If no-one creates an issue, it will not be resolved.

Peet3kabo commented 1 year ago

Just reverted back to corr 2022.6.7 and now I can't reach my file system anymore. I have this for a long time now. Everytime I revert back to an old core version the whole config/media etc folders aren't reachable anymore. I have to fully install ha again on my nvme and do a full restore... This is so anoying.

The whole config cant be reached everytime i revert back

Had the same problem when going back to 2022.6.7 but re-install studio code server did it for me. Then had al the configuration back to the old situation. Nice? No but fixable.....yes. Same with all the updates, most of the time no problems. Only twice had to go back to a previous backup. Now just waiting for a fix of the OTGW before updating. That's why you make back-ups and this is just a minor problem you have to swallow, that's what you get with free sofware. It's free because lot's of people put their time into it without asking a price for their time.

frenck commented 1 year ago

Had the same problem when going back to 2022.6.7 but re-install studio code server did it for me.

Again, same answer as above. It is not related to this issue at all, please raise an issue for backup restoring if you experience that.

Peet3kabo commented 1 year ago

You are right, not trying tot raise that as An issue. So, appoligies for the littering of the topic then

jan-gerard commented 1 year ago

I see there is a fix in an updated version of OTGW. Is there a way to manually install this update of this Integration?

frenck commented 1 year ago

@jan-gerard Only by running a dev version yourself.

This change is planned to be released with 2022.7.6 (which is the next upcoming patch release).

sygys commented 1 year ago

Thanks for the update frenck. We will have to wait for it to release then.

annegerben commented 1 year ago

@frenck I don't understand how these things can still happen. Every updates something essential breaks

If no one tests it during beta, it will be hard to achieve that. Feel free to join or beta tests to help prevent these issues.

But instead HA updates should keep all integrations intact and be backward compatible

This is upstream breaking (Python in combination with the upstream package), not Home Assistant. You are mixing up things.

To end this with a positive note, this thread was the final push I needed to also install the dev version. I hope I can help with testing in future!

umtauscher commented 1 year ago

even after 2022.7.6 OTGW isn‘t running properly. It simply doesn‘t refresh the values. It seems after the initial connect it simply stops.

farmio commented 1 year ago

@umtauscher Please open a new issue for new problems - containing details like debug logs or diagnostics information.

Asinke commented 1 year ago

Got it all working correctly in the latest 2022.7 release, however when upgrading to 2022.8.0 opentherm is misbehaving again. Sensors become available and within seconds go unavailable again all the time.

Log information:

Logger: homeassistant Source: util/async_.py:141 First occurred: 3 August 2022 at 18:55:22 (14758 occurrences) Last logged: 07:32:56

Error doing job: Exception in callback SerialTransport._call_connection_lost(SerialExcepti...disconnected')) Error doing job: Exception in callback SerialTransport._call_connection_lost(None) Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.10/site-packages/serial_asyncio/init.py", line 417, in _call_connection_lost self._serial.close() File "/usr/local/lib/python3.10/site-packages/serial/urlhandler/protocolsocket.py", line 104, in close time.sleep(0.3) File "/usr/src/homeassistant/homeassistant/util/async.py", line 180, in protected_loop_func checkloop(func, strict=strict) File "/usr/src/homeassistant/homeassistant/util/async.py", line 141, in check_loop raise RuntimeError( RuntimeError: Detected blocking call to sleep inside the event loop. Use await hass.async_add_executor_job(); This is causing stability issues. Please report issue

farmio commented 1 year ago

Seems like serial_asyncio is not so async 😬

This has been an issue some time ago. See https://github.com/pyserial/pyserial-asyncio/issues/81 I wonder why that comes up again.

There is already a HA issues for other this: https://github.com/home-assistant/core/issues/67430

home-assistant / core