Emulating an end device (preferably also router) using an NCP

[fakuivan]

Does this library provide the facilities to join a network as an end device/router define endpoints, zdo and other required features to act as a regular end device? I'm interested in turning one of those tuya zigbee gateways into routers without having to flash a router-only firmware, also this could be interesting for basic OOB management of servers: stick a zigbee USB controller and make it join the network, and say, make a push button to restart the system, give uptime/temperature readings etc.

Where should I start? Are there any projects using zigpy this way?

[puddly]

Joining networks isn't something that is currently implemented.

The node API right now is abstract enough to support normal operation once you're on a network (since it doesn't assume the device you're controlling has a NWK of 0x0000) but the joining part isn't something we ever did.

You should be able to do this pretty easily with a Silicon Labs/EZSP coordinator stick, since the bellows command set exposes the APIs to join a network. You'd have to subclass ControllerApplication and probably override start_network (or create join_network?).

[fakuivan]

I see, does that mean that I should move this discussion over to the bellows repo?

by subclassing ControllerApplication do you mean bellows.zigbee.application.ControllerApplication or zigpy.application.ControllerApplication? I'm guessing the former since it implements much of the communication with the EZSP coordinator.

[puddly]

It'll get more visibility here and I think it would be useful to generalize any changes and implement it in other radio libraries too (I've gotten it to work once with ZNP and I think deCONZ can act as a router).

Regarding subclassing: yes, the former. You can find the EZSP spec here. I think you can use joinNetwork (maybe getting the PAN ID and channel/beacon with startScan(scanType=EZSP_ACTIVE_SCAN)?).

[fakuivan]

The basic joinNetwork flow is used by the bellows join command, there seems to have been some changes to the API since the CLI was deprecated, thus it needs some changes to get it working. Still, I was able to join a node as router and end device, however the onboarding process gets stuck at "Starting interview", I'm guessing because of the lack of a proper ZDO.

[MattWestb]

I think the interview is stalling then you have not setting up endpoints and clusters and also device type is needed for the system can getting the knowledge of the device (null null is not working in the Zigbee world).

[fakuivan]

So I've experimented a bit with the end device thing and got it to join my zha network. Here's the POC:

https://github.com/fakuivan/ncp-router/blob/master/main.py

So far it manages to finish the interview and join the network, this is the device page in HA:

However, the "last seen" parameter gets stuck at the time the device joined the network, as you can see in line 64 I'm keeping the connection to the NCP open, as my understanding is that if you close the connection the NCP enters like an IDLE state to save power. I haven't filled the ZDO, so this "last seen" not being updated could be caused by that or by something like an internal watchdog not being fed.

[fakuivan]

After experimenting a bit more with the script I came to the conclusion that routing functionality is there already, at least on the surface lever. Calling permitJoining and adding some end devices results on them popping up in HA.

There are of course unhandled messages that pop up once in a while, here's an example:

Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=49, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=93), 255, -73, 0x0000, 255, 255, b';\x00']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=5, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=94), 255, -73, 0x0000, 255, 255, b'<du']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=50, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=95), 255, -73, 0x0000, 255, 255, b'=\x00']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=5, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=96), 255, -73, 0x0000, 255, 255, b'>du']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=50, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=97), 255, -73, 0x0000, 255, 255, b'?\x0f']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_UNICAST: 0>, EmberApsFrame(profileId=0, clusterId=5, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_RETRY|APS_OPTION_ENABLE_ROUTE_DISCOVERY: 320>, groupId=0, sequence=98), 255, -73, 0x0000, 255, 255, b'@du']
Got callback 'incomingMessageHandler' with params=[<EmberIncomingMessageType.INCOMING_BROADCAST: 4>, EmberApsFrame(profileId=0, clusterId=1, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_ENABLE_ROUTE_DISCOVERY: 256>, groupId=0, sequence=99), 255, -73, 0x0000, 255, 255, b'\xd4\x7fO\x00\x00']
Got callback 'messageSentHandler' with params=[<EmberOutgoingMessageType.OUTGOING_BROADCAST: 6>, 65533, EmberApsFrame(profileId=0, clusterId=1, sourceEndpoint=0, destinationEndpoint=0, options=<EmberApsOption.APS_OPTION_ENABLE_ROUTE_DISCOVERY: 256>, groupId=0, sequence=99), 0, <EmberStatus.SUCCESS: 0>, b'']

so it'd be a matter of adding missing endpoints and message handlers (like in ControllerApplication.ezsp_callback_handler and ControllerApplication.register_endpoints) for it to work on a basic level. After that I think it shouldn't be that hard to add endpoints inputs and output clusters for switches/buttons or other user defined stuff.

Edit: about HA not updating the "last seen" field in my previous comment, the device seems to work just fine without having to update a watchdog or something like that (this device is ezsp v8), as seen in ControllerApplication._watchdog_feed. My guess is that the field isn't being updated because HA isn't really liking the completely unconfigured device :P

[Hedda]

FYI, read update from @fakuivan in https://github.com/fakuivan/orvibo-gynoid-zigbee-hub-hack/issues/1 that he has now come up with a partial non-zigpy based host application workaround for his use case to make EmberZNet NCP work as a stand-alone Zigbee Router via EZSP:

• https://github.com/fakuivan/ezsp-router

Would be cool if zigpy could do it so can configure Home Assistant's ZHA integration as Zigbee Router in a other Zigbee network.

A more zigpy specific real-world scenario would be a user that has more than one instance of Home Assistant running at home., like those users that are trying to run Home Assistant in some kind of high-available setup with automatic failover.

So a user could for example have two Home Assistant Yellow appliances (or two Home Assistant Green appliances with SkyConnect dongles) running then both could be online at the same however one act as a Zigbee Coordinator and the other acts as a Zigbee Router during normal operation, but if the one that act as a Zigbee Coordinator stop responding for more than a set time then the one acting as a Zigbee Router will be reconfigured and restored as as the Zigbee Coordinator .

[MattWestb]

@Hedda Only one problem is ARG ref=gsdk_4.0 that have dropping all EMxxx and most MG1 chips (some IKEA MG1P can still being done with GSDK > 3.x but likely not this) !!!

[fakuivan]

I haven't tried changing the GSDK version, but it shouldn't be that hard to get different versions working with that Docker ARG.

As for the active backup case, I'm not sure how that's be much different from a cold spare backup, unless the standard has features for promoting a router into a controller, you'd still need to manage the Trust Center being in the old coordinator (unless you make your trust center a node different from the coordinator?). It'd be awesome if you could provide more info on that.

The POC using bellows has the right NWK stuff as far as I understand, it's a matter of adding the APL features, most of which seems to be implemented in zigpy.application.ControllerApplication. Right now I'm doing research on that part of ZigBee to add it to the GSDK C program. The main reason I haven't continued with the zigpy version is that I can't fit that program into a router CPU :/

[MattWestb]

For your EM3581 you need GSDK 3.2.x and its not on GitHub then its locked / payed licence plus its need very expensive compiler that you need paying for to using (normally they using GDB / GNU that is free).

[fakuivan]

@MattWestb ezsp-router doesn't need a compiler for the ZigBee chip since it runs on the Hub Realtek SoC, and communicates via EZSP, unless you're looking to program this on the chip directly (not the point of ezsp-router). You're welcome to continue this discussion on https://github.com/fakuivan/orvibo-gynoid-zigbee-hub-hack/issues/1 , since this is getting a bit off topic from zigpy.

[Hedda]

FYI, Nerivec has added commands to enable Zigbee Router mode (with join, rejoing and leave command options well as sniff and other utils) to his new CLI tool called ”ember-zli” for Silicon Labs EmberZNet (which depends on the ”ember” adapter in the zigbee-herdsman project). Check it out here:

• https://github.com/Nerivec/ember-zli

• Add router command

  • Join network
  • Rejoin network
  • Leave network
  • Backup NVM3 tokens
  • Restore NVM3 tokens
  • Reset NVM3 tokens
  • Get network info
  • Set manufacturer code
  • Read counters
  • Ping coordinator
  • Reload custom event handlers
  • Run custom script

• Add sniff command

  • Start sniffing

• Add utils command

  • Parse NVM3 tokens backup file

• Remove menus better suited for utils command.

  • stack

  • Get NVM3 tokens info (details of what Backup saves)