[REQUEST] nRF5340 (nRF53 series) support with pre-compiled ZBOSS NCP Host firmware image for nRF5340 DK?

Hedda commented 3 months ago

Originally posted by @JohnConnett in https://github.com/kardia-as/zigpy-zboss/discussions/43 under discussions section https://github.com/kardia-as/zigpy-zboss/discussions

I have an nRF5340 DK. Has anyone successfully used zigpy-zboss with one of these? I presume I would have to compile the ZBOSS NCP Host firmware image myself configured to use maximum memory?

Is nRF5340 supported? Can you please provide a compiled ZBOSS NCP Host firmware image required to be flash on nRF5340 DK?

https://github.com/kardia-as/nrf-zboss-ncp

https://github.com/kardia-as/zigpy-zboss?tab=readme-ov-file#firmware

https://www.nordicsemi.com/Products/Development-hardware/nRF5340-DK

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/zigbee/ncp/README.html

For reference;

nRF5340 is the latest SoC from Nordic Semiconductor with Zigbee support (combining all features of nRF52840 and nRF52833):

https://www.nordicsemi.com/products/nrf5340

It features 1 MB Flash + 512 kB RAM, and it provides better RX sensitivity (less sensitive to EMF/RMI noise) while using less power.

https://www.nordicsemi.com/-/media/Software-and-other-downloads/Product-Briefs/nRF5340-SoC-PB.pdf

https://blog.nordicsemi.com/getconnected/why-does-the-nrf5340-have-two-cores

"The 0 dBm TX current is 3.4 mA, while the RX current is only 2.7 mA, resulting in a reduction of 29% and 41% when comparing to the nRF52840 SoC. The RX sensitivity is -98 dBm, 3 dB better than nRF52840, meaning that the nRF5340 provides 3 dB better sensitivity, while using 41% less current."

https://infocenter.nordicsemi.com/index.jsp?topic=%2Fps_nrf5340%2Fkeyfeatures_html5.html

https://www.nordicsemi.com/Products/Wireless/Zigbee

https://www.nordicsemi.com/Products/Development-software/nrf5-sdk-for-thread-and-zigbee

Other than using about half the power, another reason why you might want to add support for it is for its nRF5340-DK development kit:

https://www.nordicsemi.com/Products/Development-hardware/nrf5340-dk

Ebyte/cdEbyte have an E83-2G4M03S SMD radio module as well which can be found on their E83-2G4M03S-TB evaluation board

https://ebyteiot.com/products/nrf5340-wireless-mesh-bluetooth-test-board-usb-interface-cdebyte-e83-2g4m03s-tb-easy-to-develop-bluetooth-test-kit-ipex-antenna

https://www.cdebyte.com/products/E83-2G4M03S

https://www.cdebyte.com/products/E83-2G4M03S-TB

https://www.amazon.co.uk/EBYTE-nRF5340-Wireless-Bluetooth-E83-2G4M03S-TB/dp/B0CKQGX9M2

https://www.aliexpress.com/item/1005006108259017.html

https://www.alibaba.com/product-detail/EBYTE-E83-2G4M03S-TB-Small-size_1600959383696.html

https://www.alibaba.com/product-detail/COJXU-E83-2G4M03S-Small-size-and_1600959538223.html

For commercial and DIY projects there is a nRF5340 based radio modules (BT40F, BT40, BT40E, BT40N) from Fanstel Corp.

https://www.fanstel.com/bt40f-nrf5340

https://www.fanstel.com/bc40m-nrf5340

and u-blox

https://www.fanstel.com/buy/bt840f-v1-nrf52840-bluetooth-5-thread-zigbee-module-hscxm-n7trh

https://www.u-blox.com/en/search?query=nRF5340

DamKast commented 2 months ago

Hey,

I built the NCP sample with max memory config for the nRF5340DK. I don't have the device so I can not test. @JohnConnett do you have the possibility to test it?

I uploaded the .hex file in this repo: https://github.com/kardia-as/nrf-zboss-ncp

JohnConnett commented 1 month ago

I built the NCP sample with max memory config for the nRF5340DK. I don't have the device so I can not test. @JohnConnett do you have the possibility to test it?

Many thanks. It would be useful to know what needs to be changed to build with max memory (I have nRF Connect SDK v2.6.1 installed).

I uploaded the .hex file in this repo: https://github.com/kardia-as/nrf-zboss-ncp

I have flashed the nRF5340 DK with v2.6.0_ncp_zboss_max_mem_nrf5340_usb.hex using nRF Connect for Desktop v5.0.0 and Programmer v4.3.0 using the USB connector on the end of the board.

I have also flashed the nRF52840 Dongle with v2.6.0_ncp_zboss_max_mem_nrf52840_dongle.hex using the same software.

After swapping to the USB connector on the side of the nRF5340 DK both devices have the same identification:

$ lsusb
[...]
Bus 003 Device 008: ID 2fe3:0001 NordicSemiconductor Zigbee NCP
Bus 003 Device 011: ID 2fe3:0001 NordicSemiconductor Zigbee NCP
[...]
$

When used with a Home Assistant VM with the USB device passed through I was unable to find support for the radio when installing ZHA. Is some additional configuration required? Happy to try to test this further.

JohnConnett commented 1 month ago

I think there may be problems with both v2.6.0_ncp_zboss_max_mem_nrf5340_uart.hex and v2.6.0_ncp_zboss_max_mem_nrf5340_usb.hex. When flashed, neither of them generate an entry in /dev/serial/by-id.

Using a nRF52840 Dongle flashed with v2.6.0_ncp_zboss_max_mem_nrf52840_dongle.hex:

$ ls -l /dev/serial/by-id
total 0
lrwxrwxrwx 1 root root 13 May 15 15:35 usb-ZEPHYR_Zigbee_NCP_C90A32EC1283F3AC-if00 -> ../../ttyACM0
$

I don't have access to a nRF52840 DK. If someone has successfully used one it would be interesting to know how they achieved it. Hopefully, the nRF52840 DK and nRF5340 DK would be broadly similar.

I have set up a development environment for Developing with Visual Studio Code + devcontainer and added zigpy-zboss support roughly as described in How to get started with zigpy-zboss and Home Assistant?.

The missing piece is how to pass through /dev/ttyACM0 from the host to the devcontainer ...

Hedda commented 1 month ago

The missing piece is how to pass through /dev/ttyACM0 from the host to the devcontainer ...

If you are using Docker then you to install and use something like Docker Compose to forward your device from the host machine to the Docker instance, see:

https://www.home-assistant.io/integrations/zha#cant-connect-to-usb-device-and-using-docker

also check that the the USB or serial device is not being claimed other services on the host, see example.

https://www.home-assistant.io/integrations/zha#zha-start-up-issue-with-home-assistant-or-home-assistant-container

PS: I do not have much experince with Docker and other container systems.

JohnConnett commented 1 month ago

The missing piece is how to pass through /dev/ttyACM0 from the host to the devcontainer ...

If you are using Docker then you to install and use something like Docker Compose to forward your device from the host machine to the Docker instance, see:

https://www.home-assistant.io/integrations/zha#cant-connect-to-usb-device-and-using-docker

I have installed docker onto Ubuntu 24.04 LTS (Noble Numbat) following Install using the apt repository. This includes installing the docker-compose-plugin package.

also check that the the USB or serial device is not being claimed other services on the host, see example.

https://www.home-assistant.io/integrations/zha#zha-start-up-issue-with-home-assistant-or-home-assistant-container

I don't think anything is claiming /dev/ttyACM0:

$ sudo lsof /dev/ttyACM0
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.

PS: I do not have much experience with Docker and other container systems.

Probably more than me!

I suspect a docker.compose.yml file needs to be added to the devcontainer, perhaps something like:

$ cat docker-compose.yml
version: '2'
services:
  myservice:
    build:
      context: /workspaces/core
      dockerfile: Dockerfile
    image: result/latest
    devices:
      # your usb device forwarding to the docker image
      - /dev/ttyACM0:/dev/ttyACM0
$

However, I don't know what to do next ...

JohnConnett commented 1 month ago

Tried another approach:

diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index 362d4cbd02..7ea3ef3bd6 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -13,7 +13,7 @@
   },
   // Port 5683 udp is used by Shelly integration
   "appPort": ["8123:8123", "5683:5683/udp"],
-  "runArgs": ["-e", "GIT_EDITOR=code --wait"],
+  "runArgs": ["-e", "GIT_EDITOR=code --wait", "--device=/dev/ttyACM0:/dev/ttyACM0"],
   "customizations": {
     "vscode": {
       "extensions": [

Unfortunately, that hits problems when trying to add the ZHA integration:

TypeError: can't convert undefined to object
entries (src/panels/config/integrations/dialog-add-integration.ts:191:13)
apply (node_modules/memoize-one/dist/memoize-one.esm.js:37:34)
_getIntegrations (src/panels/config/integrations/dialog-add-integration.ts:293:16)
_maybeSubmit (src/panels/config/integrations/dialog-add-integration.ts:671:30)
call (src/lit-html.ts:2018:28)

Any suggestions?

JohnConnett commented 1 month ago

Strange. Tried it again and the ZHA integration was found and I was able to select the ZBOSS radio. However, this failed with:

2024-05-17 09:39:45.151 INFO (SyncWorker_2) [homeassistant.loader] Loaded zha from homeassistant.components.zha
2024-05-17 09:39:45.343 INFO (ImportExecutor_0) [zigpy.serial] Using pyserial-asyncio-fast in place of pyserial-asyncio
2024-05-17 09:40:10.789 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/dev/ttyACM0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/ban.py", line 88, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp_session/__init__.py", line 199, in factory
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/auth.py", line 295, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/decorators.py", line 71, in with_admin
    return await func(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/config/config_entries.py", line 222, in post
    return await super().post(request, flow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/data_validator.py", line 73, in wrapper
    return await method(view, request, data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/helpers/data_entry_flow.py", line 122, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 367, in async_configure
    result = await self._async_configure(flow_id, user_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 413, in _async_configure
    result = await self._async_handle_step(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 517, in _async_handle_step
    result: _FlowResultT = await getattr(flow, method)(user_input)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/zha/config_flow.py", line 280, in async_step_manual_port_config
    if await self._radio_mgr.radio_type.controller.probe(user_input):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/zigbee/application.py", line 491, in probe
    await zboss.connect()
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/api.py", line 79, in connect
    self._uart = await uart.connect(
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/uart.py", line 280, in connect
    _, protocol = await zigpy.serial.create_serial_connection(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy/serial.py", line 48, in create_serial_connection
    transport, protocol = await pyserial_asyncio.create_serial_connection(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/serial_asyncio_fast/__init__.py", line 508, in create_serial_connection
    serial_instance = await loop.run_in_executor(None, callback)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/__init__.py", line 90, in serial_for_url
    instance.open()
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 2] could not open port /dev/ttyACM0: [Errno 2] No such file or directory: '/dev/ttyACM0'

Looks like /dev/ttyACM0 isn't being passed through ...

JohnConnett commented 1 month ago

Exited VS Code. Restarted VS Code and was prompted to rebuild the devcontainer. Clicked the "rebuild" button.

The ZHA integration spotted that there was a suitable device on /dev/ACM0. Went as far as "Serial Port Settings" but when the "Submit" button was clicked "Unknown error occurred" was reported. This was on the VS Code terminal:

2024-05-17 09:53:27.930 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of zigpy-zboss==1.1.5
2024-05-17 09:53:29.590 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of zigpy-zigate==0.12.0
2024-05-17 09:53:31.792 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of zigpy-znp==0.12.1
2024-05-17 09:53:33.457 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of universal-silabs-flasher==0.0.18
2024-05-17 09:53:35.508 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of pyserial-asyncio-fast==0.11
2024-05-17 09:53:37.381 INFO (ImportExecutor_0) [zigpy.serial] Using pyserial-asyncio-fast in place of pyserial-asyncio
2024-05-17 09:54:10.239 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.240 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.241 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.242 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.242 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.244 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.245 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by integration 'zha' at homeassistant/components/zha/radio_manager.py, line 221: probe_result = await radio.controller.probe(dev_config) (offender: /home/vscode/.local/lib/python3.12/site-packages/serial/tools/list_ports_linux.py, line 84: with open(os.path.join(*args)) as f:), please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+zha%22
2024-05-17 09:54:10.248 INFO (MainThread) [universal_silabs_flasher.flasher] Probing ApplicationType.GECKO_BOOTLOADER at 115200 baud
2024-05-17 09:54:21.859 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/dev/ttyACM0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/ban.py", line 88, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp_session/__init__.py", line 199, in factory
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/auth.py", line 295, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/decorators.py", line 71, in with_admin
    return await func(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/config/config_entries.py", line 222, in post
    return await super().post(request, flow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/http/data_validator.py", line 73, in wrapper
    return await method(view, request, data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/helpers/data_entry_flow.py", line 122, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 367, in async_configure
    result = await self._async_configure(flow_id, user_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 413, in _async_configure
    result = await self._async_handle_step(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/data_entry_flow.py", line 517, in _async_handle_step
    result: _FlowResultT = await getattr(flow, method)(user_input)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/core/homeassistant/components/zha/config_flow.py", line 280, in async_step_manual_port_config
    if await self._radio_mgr.radio_type.controller.probe(user_input):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/zigbee/application.py", line 491, in probe
    await zboss.connect()
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/api.py", line 79, in connect
    self._uart = await uart.connect(
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy_zboss/uart.py", line 280, in connect
    _, protocol = await zigpy.serial.create_serial_connection(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/zigpy/serial.py", line 48, in create_serial_connection
    transport, protocol = await pyserial_asyncio.create_serial_connection(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/serial_asyncio_fast/__init__.py", line 508, in create_serial_connection
    serial_instance = await loop.run_in_executor(None, callback)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/__init__.py", line 90, in serial_for_url
    instance.open()
  File "/home/vscode/.local/lib/python3.12/site-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 13] could not open port /dev/ttyACM0: [Errno 13] Permission denied: '/dev/ttyACM0'

It looks like /dev/ttyACM0 is now being passed through but does not have the correct permissions. On the host system:

$ ls -l /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 May 17 10:38 /dev/ttyACM0
$

JohnConnett commented 1 month ago

Swapped the nRF52840 Dongle for a SkyConnect; edited devcontainer/devcontainer.json to replace /dev/ACM0 with /dev/USB0; rebuilt the devcontainer and ran again.

Home Assistant spotted the SkyConnect but objected "unsupported firmware". Checked for Firmware Updates for SkyConnect. It had the latest firmware Zigbee (EZSP) 7.3.2.0 build 212 but re-flashed it just to be sure.

No firmware issues detected on the next attempt. Went as far as "Serial Port Settings" but when the "Submit" button was clicked "Failed to connect" was reported.

If it isn't working with the SkyConnect, I suspect there is something wrong with my attempt to configure the development environment ...

JohnConnett commented 1 month ago

If anyone wants to try to reproduce the problems with the nRF52840 Dongle, here are all of my diffs:

diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
index 362d4cbd02..7ea3ef3bd6 100644
--- a/.devcontainer/devcontainer.json
+++ b/.devcontainer/devcontainer.json
@@ -13,7 +13,7 @@
   },
   // Port 5683 udp is used by Shelly integration
   "appPort": ["8123:8123", "5683:5683/udp"],
-  "runArgs": ["-e", "GIT_EDITOR=code --wait"],
+  "runArgs": ["-e", "GIT_EDITOR=code --wait", "--device=/dev/ttyACM0:/dev/ttyACM0"],
   "customizations": {
     "vscode": {
       "extensions": [
diff --git a/homeassistant/components/zha/core/const.py b/homeassistant/components/zha/core/const.py
index 74110d390e..c6cfb68d80 100644
--- a/homeassistant/components/zha/core/const.py
+++ b/homeassistant/components/zha/core/const.py
@@ -11,6 +11,7 @@ import zigpy.application
 import zigpy.types as t
 import zigpy_deconz.zigbee.application
 import zigpy_xbee.zigbee.application
+import zigpy_zboss.zigbee.application
 import zigpy_zigate.zigbee.application
 import zigpy_znp.zigbee.application

@@ -196,6 +197,7 @@ DEBUG_COMP_ZIGPY = "zigpy"
 DEBUG_COMP_ZIGPY_ZNP = "zigpy_znp"
 DEBUG_COMP_ZIGPY_DECONZ = "zigpy_deconz"
 DEBUG_COMP_ZIGPY_XBEE = "zigpy_xbee"
+DEBUG_COMP_ZIGPY_ZBOSS = "zigpy_zboss"
 DEBUG_COMP_ZIGPY_ZIGATE = "zigpy_zigate"
 DEBUG_LEVEL_CURRENT = "current"
 DEBUG_LEVEL_ORIGINAL = "original"
@@ -206,6 +208,7 @@ DEBUG_LEVELS = {
     DEBUG_COMP_ZIGPY_ZNP: logging.DEBUG,
     DEBUG_COMP_ZIGPY_DECONZ: logging.DEBUG,
     DEBUG_COMP_ZIGPY_XBEE: logging.DEBUG,
+    DEBUG_COMP_ZIGPY_ZBOSS: logging.DEBUG,
     DEBUG_COMP_ZIGPY_ZIGATE: logging.DEBUG,
 }
 DEBUG_RELAY_LOGGERS = [DEBUG_COMP_ZHA, DEBUG_COMP_ZIGPY]
@@ -271,6 +274,10 @@ class RadioType(enum.Enum):
         "XBee = Digi XBee Zigbee radios: Digi XBee Series 2, 2C, 3",
         zigpy_xbee.zigbee.application.ControllerApplication,
     )
+    zboss = (
+        "ZBOSS = Nordic ZBOSS Zigbee radios: nRF52840, nrf5340",
+        zigpy_zboss.zigbee.application.ControllerApplication,
+    )

     @classmethod
     def list(cls) -> list[str]:
diff --git a/homeassistant/components/zha/manifest.json b/homeassistant/components/zha/manifest.json
index 9a0ca62542..fabd8c8fdc 100644
--- a/homeassistant/components/zha/manifest.json
+++ b/homeassistant/components/zha/manifest.json
@@ -16,6 +16,7 @@
     "zigpy",
     "zigpy_deconz",
     "zigpy_xbee",
+    "zigpy_zboss",
     "zigpy_zigate",
     "zigpy_znp",
     "universal_silabs_flasher"
@@ -27,6 +28,7 @@
     "zigpy-deconz==0.23.1",
     "zigpy==0.64.0",
     "zigpy-xbee==0.20.1",
+    "zigpy-zboss==1.1.5",
     "zigpy-zigate==0.12.0",
     "zigpy-znp==0.12.1",
     "universal-silabs-flasher==0.0.18",

Hedda commented 1 month ago

Probably does not help but FYI, there are some additional tips regarding installation and configuration in these zigpy documents:

Hedda commented 1 month ago

@JohnConnett Can I suggest that you try running in a VM (virtual machine) under a virtualisation hypervisor platform instead?

https://en.wikipedia.org/wiki/Hypervisor

Then you have the option to run directly under a full operating system as well as bonus features such as snapshots.

In my experience it is much easier to configure USB passthrough to devices for VM hardware in a virtual machine manager.

I run VMs under VMware Workstation/Fusion/ESXi myself but many alternatives to those can be used free of charge, ex:

And several other less common alternatives -> https://en.wikipedia.org/wiki/Comparison_of_platform_virtualization_software

puddly commented 1 month ago

@JohnConnett Your user (or the Docker user) likely does not have permission to access serial ports because they aren't part of the dialout group.

You may also find using Home Assistant Core directly a simpler development experience than the dev container: https://developers.home-assistant.io/docs/development_environment/#setup-local-repository. It is just a Python package after all.

JohnConnett commented 1 month ago

Looks like I missed a few of the changes needed to support ZBOSS. Here's my latest diff:

diff --git a/homeassistant/components/zha/core/const.py b/homeassistant/components/zha/core/const.py
index 74110d390e..c6cfb68d80 100644
--- a/homeassistant/components/zha/core/const.py
+++ b/homeassistant/components/zha/core/const.py
@@ -11,6 +11,7 @@ import zigpy.application
 import zigpy.types as t
 import zigpy_deconz.zigbee.application
 import zigpy_xbee.zigbee.application
+import zigpy_zboss.zigbee.application
 import zigpy_zigate.zigbee.application
 import zigpy_znp.zigbee.application

@@ -196,6 +197,7 @@ DEBUG_COMP_ZIGPY = "zigpy"
 DEBUG_COMP_ZIGPY_ZNP = "zigpy_znp"
 DEBUG_COMP_ZIGPY_DECONZ = "zigpy_deconz"
 DEBUG_COMP_ZIGPY_XBEE = "zigpy_xbee"
+DEBUG_COMP_ZIGPY_ZBOSS = "zigpy_zboss"
 DEBUG_COMP_ZIGPY_ZIGATE = "zigpy_zigate"
 DEBUG_LEVEL_CURRENT = "current"
 DEBUG_LEVEL_ORIGINAL = "original"
@@ -206,6 +208,7 @@ DEBUG_LEVELS = {
     DEBUG_COMP_ZIGPY_ZNP: logging.DEBUG,
     DEBUG_COMP_ZIGPY_DECONZ: logging.DEBUG,
     DEBUG_COMP_ZIGPY_XBEE: logging.DEBUG,
+    DEBUG_COMP_ZIGPY_ZBOSS: logging.DEBUG,
     DEBUG_COMP_ZIGPY_ZIGATE: logging.DEBUG,
 }
 DEBUG_RELAY_LOGGERS = [DEBUG_COMP_ZHA, DEBUG_COMP_ZIGPY]
@@ -271,6 +274,10 @@ class RadioType(enum.Enum):
         "XBee = Digi XBee Zigbee radios: Digi XBee Series 2, 2C, 3",
         zigpy_xbee.zigbee.application.ControllerApplication,
     )
+    zboss = (
+        "ZBOSS = Nordic ZBOSS Zigbee radios: nRF52840, nrf5340",
+        zigpy_zboss.zigbee.application.ControllerApplication,
+    )

     @classmethod
     def list(cls) -> list[str]:
diff --git a/homeassistant/components/zha/core/gateway.py b/homeassistant/components/zha/core/gateway.py
index 009364ba9d..26f5f0cedf 100644
--- a/homeassistant/components/zha/core/gateway.py
+++ b/homeassistant/components/zha/core/gateway.py
@@ -56,6 +56,7 @@ from .const import (
    DEBUG_COMP_ZIGPY,
     DEBUG_COMP_ZIGPY_DECONZ,
     DEBUG_COMP_ZIGPY_XBEE,
+    DEBUG_COMP_ZIGPY_ZBOSS,
     DEBUG_COMP_ZIGPY_ZIGATE,
     DEBUG_COMP_ZIGPY_ZNP,
     DEBUG_LEVEL_CURRENT,
@@ -833,6 +834,9 @@ def async_capture_log_levels() -> dict[str, int]:
         DEBUG_COMP_ZIGPY_XBEE: logging.getLogger(
             DEBUG_COMP_ZIGPY_XBEE
         ).getEffectiveLevel(),
+        DEBUG_COMP_ZIGPY_ZBOSS: logging.getLogger(
+            DEBUG_COMP_ZIGPY_ZBOSS
+        ).getEffectiveLevel(),
         DEBUG_COMP_ZIGPY_ZIGATE: logging.getLogger(
             DEBUG_COMP_ZIGPY_ZIGATE
         ).getEffectiveLevel(),
@@ -848,6 +852,7 @@ def async_set_logger_levels(levels: dict[str, int]) -> None:
     logging.getLogger(DEBUG_COMP_ZIGPY_ZNP).setLevel(levels[DEBUG_COMP_ZIGPY_ZNP])
     logging.getLogger(DEBUG_COMP_ZIGPY_DECONZ).setLevel(levels[DEBUG_COMP_ZIGPY_DECONZ])
     logging.getLogger(DEBUG_COMP_ZIGPY_XBEE).setLevel(levels[DEBUG_COMP_ZIGPY_XBEE])
+    logging.getLogger(DEBUG_COMP_ZIGPY_ZBOSS).setLevel(levels[DEBUG_COMP_ZIGPY_ZBOSS])
     logging.getLogger(DEBUG_COMP_ZIGPY_ZIGATE).setLevel(levels[DEBUG_COMP_ZIGPY_ZIGATE])

diff --git a/homeassistant/components/zha/manifest.json b/homeassistant/components/zha/manifest.json
index 9a0ca62542..fabd8c8fdc 100644
--- a/homeassistant/components/zha/manifest.json
+++ b/homeassistant/components/zha/manifest.json
@@ -16,6 +16,7 @@
     "zigpy",
     "zigpy_deconz",
     "zigpy_xbee",
+    "zigpy_zboss",
     "zigpy_zigate",
     "zigpy_znp",
     "universal_silabs_flasher"
@@ -27,6 +28,7 @@
     "zigpy-deconz==0.23.1",
     "zigpy==0.64.0",
     "zigpy-xbee==0.20.1",
+    "zigpy-zboss==1.1.5",
     "zigpy-zigate==0.12.0",
     "zigpy-znp==0.12.1",
     "universal-silabs-flasher==0.0.18",

The good news is that with these changes applied running them on Ubuntu 24.04 LTS (Noble Numbat) with the nRF52840 Dongle attached with hass -c config was sufficient to create a Zigbee network and add a single Philips Hue light.

Unfortunately, this didn't fix the permissions problem when running in the devcontainer. I also tried using --privileged instead of --device=/dev/ttyACM0:/dev/ttyACM0 but hit a similar permissions problem.

JohnConnett commented 1 month ago

@JohnConnett Your user (or the Docker user) likely does not have permission to access serial ports because they aren't part of the dialout group.

My user is in the dialout group, the Docker user may not be. I also tried with --privileged and still had permission problems with /dev/ttyACM0.

You may also find using Home Assistant Core directly a simpler development experience than the dev container: https://developers.home-assistant.io/docs/development_environment/#setup-local-repository. It is just a Python package after all.

Yes, that worked for me.

Looks like it is possible to use the nRF52840 Dongle. Need to figure out if the nRF5340 DK can be made to work. It has two micro USB connectors so I'm not sure which of them needs to be connected. My guess is that the one on the end is for SEGGER and the one on the side is for UART.

DamKast commented 1 week ago

It has two micro USB connectors so I'm not sure which of them needs to be connected. My guess is that the one on the end is for SEGGER and the one on the side is for UART.

@JohnConnett It will depend on which firmware build is flashed on your DK. Check this out: https://github.com/kardia-as/nrf-zboss-ncp/tree/main/nRF5340DK

kardia-as / zigpy-zboss

[REQUEST] nRF5340 (nRF53 series) support with pre-compiled ZBOSS NCP Host firmware image for nRF5340 DK? #46