ZHA unable to find serial connection after b2 upgrade

[Madj42]

The problem

Immediately after the the upgrade to beta 2, zigbee devices that are managed via the ZHA integration work and respond. When you restart core, they fall to come up and are unavailable. A restart of the device does not fix the issue. Downgrading back to beta 1 with or without a restart fixes the issue. While this appears to be a stick issue, the fact that it works after downgrading says otherwise. Have not had an issue until updating to this beta. I'm running HassOS 5.10 64bit on a RPI 4.

What is version of Home Assistant Core has the issue?

Core-2021.2.0b2

What was the last working version of Home Assistant Core?

Core-2021.2.0b1

What type of installation are you running?

Home Assistant OS

Integration causing the issue

ZHA

Link to integration documentation on our website

https://www.home-assistant.io/integrations/zha/

Example YAML snippet

# Put your YAML below this line

Anything in the logs that might be useful for us?

# Put your logs below this line

Logger: homeassistant.components.zha.core.gateway
Source: components/zha/core/gateway.py:157
Integration: Zigbee Home Automation (documentation, issues)
First occurred: 10:09:00 PM (7 occurrences)
Last logged: 10:12:56 PM

Couldn't start deCONZ = dresden elektronik deCONZ protocol: ConBee I/II, RaspBee I/II coordinator
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2196260-if00'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/zha/core/gateway.py", line 157, in async_initialize
    self.application_controller = await app_controller_cls.new(
  File "/usr/local/lib/python3.8/site-packages/zigpy/application.py", line 69, in new
    await app.startup(auto_form)
  File "/usr/local/lib/python3.8/site-packages/zigpy_deconz/zigbee/application.py", line 65, in startup
    await self._api.connect()
  File "/usr/local/lib/python3.8/site-packages/zigpy_deconz/api.py", line 245, in connect
    self._uart = await zigpy_deconz.uart.connect(self._config, self)
  File "/usr/local/lib/python3.8/site-packages/zigpy_deconz/uart.py", line 139, in connect
    _, protocol = await serial_asyncio.create_serial_connection(
  File "/usr/local/lib/python3.8/site-packages/serial_asyncio/__init__.py", line 445, in create_serial_connection
    serial_instance = serial.serial_for_url(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/serial/__init__.py", line 90, in serial_for_url
    instance.open()
  File "/usr/local/lib/python3.8/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 2] could not open port /dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2196260-if00: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2196260-if00'

Logger: zigpy.application
Source: /usr/local/lib/python3.8/site-packages/zigpy/application.py:71
First occurred: 10:09:00 PM (7 occurrences)
Last logged: 10:12:56 PM

Couldn't start application

[probot-home-assistant[bot]]

zha documentation zha source _{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @dmulcahey, @adminiuga, mind taking a look at this issue as its been labeled with an integration (zha) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

[Adminiuga]

It cannot find the serial port device FileNotFoundError: [Errno 2] No such file or directory: '/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2196260-if00'

Without it zha would not start

[SeanPM5]

Had the same issue immediately after upgrading to core-2021.2.0b2 (and also on a Rpi 4, although with HA OS 5.8 64bit).

Restarted HA numerous times, unplugged the stick and plugged it back in, etc. Didn't work even though the stick was still being recognized in Supervisor -> System -> Hardware section.

Downgrading back to b1 fixed the issue.

[Adminiuga]

May want to open an issue in hassos repo then.

[Madj42]

I'm not sure how this would be an OS issue though with the downgrade back to b1 fixing the issue. Downgrading core fixes the issue and I'm not touching the OS or rebooting.

[Adminiuga]

Because it is a different container??? There were no zha changes since b0

[Adminiuga]

And it is not like the zha integration creating that file.

[Madj42]

I'm baffled as I cannot reproduce this any longer after trying to again. I was able to reproduce this about a dozen times after upgrading to b2 only to find now it's working. Apologize for the false alarm.

[Veldkornet]

I have this issue too after using b2. Initially it there was no problem, but this evening all of a sudden it stopped working. Can’t get it up and running again, replugged, rebooted, no success. It’s all gone to ####

[Adminiuga]

After re-plugging, you need to restart ha container, as the stick needs to be plugged in, before container is started, otherwise it won't see the device file, even though it would show up in the supervisor -- that's misleading.

[Veldkornet]

Yup, I’ve done that, restart with them plugged in. Restarted many many many many times.

Weirdly, my Aeotec ZWave stick works in the zwavejs2mqtt container perfectly, but things that are “directly” used by HomeAssistant are broken. For me that’s ZHA with my ConBee II as well as my RFXcom 433Hmz

[Veldkornet]

What’s the command in ha to downgrade to b1?

[SeanPM5]

What’s the command in ha to downgrade to b1?

ha core update --version 2021.2.0b1

[Adminiuga]

And getting the same file not found error? May want to raise an issue in hassos or is there a special repo for the home assistant docker container? Especially since you are getting the error with rx433 integration, which is same root cause and it is not the ZHA

[Veldkornet]

Isn’t this the repo for ha os? Someone just needs to remove the zha tag.

[Veldkornet]

What’s the command in ha to downgrade to b1?

ha core update --version 2021.2.0b1

Oh thank goodness, I can use my lights again

[jscherry]

same issue as above with b2 beta. Using Conbee II on HA Blue

[jscherry]

just downgraded to b1 beta and everything is working again. Not sure how this is an OS problem as both are running on 5.10 OS?

[Adminiuga]

so both .b2 and .b3 stop working? Can you confirm you get exactly the same error [Errno 2] No such file or directory: '/dev/serial/by-id/...

[jscherry]

I've already restored to b1 and had to delete the database malformed. I was getting an error No such file or directory: '/dev/serial/by-id/... I can't remember if it said Errno 2? It was stuck in a loop of removing and re-adding the usb port with the conbee. I also remember an error saying usb-port was no longer valid, change to usb or something like that.

[Adminiuga]

I could not reproduce with ConBee I on Pi3B+ HassOS 5.10 I've tried upgrading from 2020.12.x to 2021.2.0b1 .b2 and .b3

[elupus]

Nothing obvious in diff: https://github.com/home-assistant/core/compare/2021.2.0b2...2021.2.0b3

[Veldkornet]

Nothing obvious in diff: https://github.com/home-assistant/core/compare/2021.2.0b2...2021.2.0b3

Shouldn't we look between b1 and b3?

https://github.com/home-assistant/core/compare/2021.2.0b1...2021.2.0b3

[elupus]

I think we need to know what other integrations you guys have active.

[Veldkornet]

Yup, I’ve done that, restart with them plugged in. Restarted many many many many times.

Weirdly, my Aeotec ZWave stick works in the zwavejs2mqtt container perfectly, but things that are “directly” used by HomeAssistant are broken. For me that’s ZHA with my ConBee II as well as my RFXcom 433Hmz

Besides the ZHA error which is the same as the OP, my RFXcom only shows the following: 2021-02-03 01:07:04 ERROR (MainThread) [homeassistant.components.rfxtrx] Connection timeout: failed to receive response from RFXtrx device

I've now upgraded from b1 to b2 to try reproduce the error. It wasn't instant the previous time, seemed to happen after I restarted HA after some time (the Z-Wave JS to MQTT add-on was updated, so I restarted to let HA discover everything again). So, I'll follow the same route this time round too. Directly what I did see though is the RFXcom error I posted above.

If I go to "Hardware Info" on the supervisor, I also see A LOT of information (1200 lines of info). Way more than I remember seeing the last time I looked there.

I also went in via SSH and listed /dev/serial/by-id/ and confirmed that the devices do show up there.

FYI, I also use HassOS 5.10 64bit on a RPI 4

[SeanPM5]

Could you try upgrading to b3 and see whether it occurs there? b1 and b3 both work fine on my end, it was only b2 that was problematic for me.

[Veldkornet]

I tried b3 yesterday evening when it was released, and it appeared that everything was working there (although maybe it just needed more time). b3 breaks all of my zwave binary_sensors though, so I'm avoiding it for the moment...

[chmielowiec]

I could not reproduce with ConBee I on Pi3B+ HassOS 5.10 I've tried upgrading from 2020.12.x to 2021.2.0b1 .b2 and .b3

Same here on Conbee II, but I'm on Home Assistant Container without Supervisor, so it seems to be not HA Core problem.

[Adminiuga]

It could be supervisor related and depend on what version of supervisor the container was created. After upgrading the supervisor and ha core to .b2 or .b3, could you run ha core rebuild

[Veldkornet]

I think b4 will fix it with https://github.com/home-assistant/core/pull/45889

[Adminiuga]

Should be fixed by https://github.com/home-assistant/core/releases/tag/2021.2.0b4

Give it a try once it hits the docker hub

[jscherry]

I also see a lot more code info under hardware than before upgrading to b2 exactly as the comment above. "Hardware Info" on the supervisor, I also see A LOT of information (1200 lines of info). Way more than I remember seeing the last time I looked there.

[rjgrandy]

I can also confirm I saw the problem in b3 and now it is fixed in b5. I also installed zwavejs2mqtt addon in b3, just for reference.

[Adminiuga]

Considering this fixed in 2021.2.0 Release. Closing this one. Re-open if the issue persists and submit detailed error message.