Zigbee-network map

[kirovilya]

For my application, I began to make feature to display the network map. It shows the connections between devices and nodes, as well as the quality of link. I use the ZShepherd.prototype.lqiScan (thanks to @splitice and HalleyAssist fork) function that collects this info. https://github.com/kirovilya/zigbee-shepherd/blob/dev/lib/shepherd.js#L417

But it does not always work correctly and some devices are not included in the map. Until I figured out why and how to fix it. :(

Another possibility is to start the pairing process on a separate router, rather than in the entire network. To do this, you need to send the NWK-address of the device to permitJoin function.

https://github.com/kirovilya/zigbee-shepherd/blob/dev/lib/components/controller.js#L338

[clockbrain]

Good plan! I'd also be keen to see some device mapping info available via MQTT which could potentially be rendered by e.g. HA.

I got a map of my small network using the method from here https://community.smartthings.com/t/faq-mapping-your-zigbee-network-with-digis-xctu/28477/55. I could do this because I had a Digi XU-Z11 XStick laying around from when I had previously hoped it would work with HA. Here's the map it shows of my zigbee2mqtt network.

The XU-Z11 is the R in the top left. The CC2531 is the C. Note that it shows true mesh links and did pick up all my devices. Not suggesting that an XU-Z11 should be required, this is just an example of the level of detail that would be useful in a map. Does the lqi function report the mesh or just shortest path?

[kirovilya]

@clockbrain In contrast to Digi XU-Z11, this map can be obtained from coordinator and sent by mqtt to HA.

[splitice]

@kirovilya If the problem is on our end PR's are welcome.

I suspect however that the device may not be LQI robust (sometimes ignoring them or similar). Currently the LQI scan is lazy (it's intensive enough without repeats) but I would welcome any well thought out patch for robustness in any area of our forks.

[kirovilya]

@splitice yes of course! thank you!

[kirovilya]

@splitice I found one error in your code :) https://github.com/HalleyAssist/zigbee-shepherd/blob/master/lib/shepherd.js#L426 - you has not return result in chain function, in compare with my version https://github.com/kirovilya/zigbee-shepherd/blob/dev/lib/shepherd.js#L433

[splitice]

Thanks, Nice find, looks like you are correct. I'll patch that in the morning

[kirovilya]

I added more devices with 2 router. Some end-devices join to first router, some devices to second, and one to coordinator.

[Koenkk]

This looks really great!

[tb-killa]

@Koenkk what about provide json array with map Informations as mqtt topic?

[Koenkk]

@tb-killa I think we need a separate topic for this.

Also it would be nice if we somehow could provide a rendered graph to home assistant (e.g. via an http jpeg endpoint) which can be used with https://www.home-assistant.io/components/camera.generic/

[kirovilya]

I used the visjs to draw a graph.

[tb-killa]

Yes New topic sounds right. Some Javascript map to jpeg could be work. But if we would provide the source of map data via mqtt json i think it would be the best if the thirdparty application would generate this. As your example we could provide some sort of docker who use our input and generate the jpeg Web. I think we should Show that zigbee2mqtt is cleaner and doesnt overloaded :)

[tb-killa]

With visjs we could export as png... Does this is okay or have it to be jpeg?

https://stackoverflow.com/questions/42663203/export-visjs-network-to-jpeg-png-image

[kirovilya]

I think that the rendering of the image should not be on the side of the Zigbee2mqtt

[tb-killa]

@kirovilya would you be willing to submit your networkmap code as pr so we can continue with the editing?

[kirovilya]

@tb-killa ok. later, in evening

[Baseje]

Would be nice to create an Hassio plugin for installing this! Really interested in this tools!

[kirovilya]

Example for PR https://github.com/Koenkk/zigbee-shepherd/pull/3 :

shepherd.lqiScan().then((result)=>{
     debug('lqiScan result: ', result);
});

[kirovilya]

Permit join through router:

const router = shepherd._findDevByAddr(routerIeeeAddr);
const permitDev = router.nwkAddr;
shepherd.permitJoin(permitTime ? permitTime : 0, permitDev, (error) => {});

[kirovilya]

then tests are needed for the max number of connected devices. eg: 10 devices on the coordinator + 2 routers + 10 devices on the router1 and 10 devices on the router2

I have not so much :)

[splitice]

Please do pr any tests you write :)

[tb-killa]

@kirovilya doesnt we could use lqi Infos from https://github.com/Koenkk/zigbee2mqtt/issues/173??

[kirovilya]

@tb-killa yes we can, but in liqScan it also exist :)

[tb-killa]

@Koenkk what about building new timer or use pollTimer and return output of shepherd.lqiScan() on mqtt topic like networkmap or another name.

If we got this i could work on building some sort of docker image for mqtt to png or mqtt to dot to jpeg as webstream.

[Koenkk]

Does this scanning wakeup the devices? Because then we should not do it on an interval basis.

[kirovilya]

I think - no. Since the request is addressed to the coordinator and the routers, and they do not sleep.

[Koenkk]

Ok thats good! Than the thing we should do is that zigbee2mqtt periodically sends it network topology in a json structure over mqtt.

[kirovilya]

I believe that this should be on request in some topic. Because the map is not needed constantly for work.

[Koenkk]

Yes that would be even better

[Koenkk]

First implementation is done, idea is to support more types.

To request a network map, publish to zigbee2mqtt/bridge/networkmap raw. Zigbee2mqtt will respond on zigbee2mqtt/bridge/networkmap/raw (note this takes a few seconds).

[Koenkk]

Basic graphviz networkmap is also done, can be request by: zigbee2mqtt/bridge/networkmap graphviz. Zigbee2mqtt will response on zigbee2mqtt/bridge/networkmap/graphviz.

Can easily be visualized on: http://www.webgraphviz.com/

[Baseje]

In wich update?

[Koenkk]

It's currently in the dev branch.

[AnhDuc85]

nice feature, thank you I removed the router (osram plug) where most of my end device connected through. After that, my end devices are not connected to the network anymore (checked via network scan and zigbee logs). Some of them are even closer to the coordinator. I tried to restart zigbee but no luck. Seems like, if an device is connected via router, the connection is permanent and the end device does not lookup for an better route. Is that normal?

[lolorc]

nice, would it make sense to add more information to the json/graphviz responses ? (like friendly name, device type, manufacturer+model, powersource, jointime...) ?

[lolorc]

@AnhDuc85 looking at my graph, it looks I also have some weird behaviours like yours with my osram plug

[lolorc]

i've got 2 devices that don't appear in the network map. looks like a link quality issue, those are aqara sensors. they don't connect thru their nearby router, linkquality is quite low (5). Still I receive data from them and the daemon publishes them to mqtt. weird.

edit: also weird, a router I stopped 12h ago is still seen as online.

[lolorc]

mainly aqara devices, prise_* are osram plugs

really weird:

• coordinator cc2531_2 is in same room as prise_01, 2 meters away, line of sight, linkquality reported by zigbee2mqtt is around 52, i was not expecting prise_01 to be routed by prise_02 (linkquality 15 in mqtt msgs)
• prise_04 and prise_02 are 15cm away from each other, same powerstrip, 2 floors higher than router
• aqara_03 and aqara_01 are on the floor than prise_04/02 few meters away, I'm expecting them to be routed...
• cube is in configuration.yaml but has not been paired to the router

[Koenkk]

Yes, graphviz is still WIP, more things will be added (like friendly name, device model, etc.)

[AnhDuc85]

Got my network map, where most of my devices were connected to an Osram plug. After that, I removed the osram plug from my network. Some hours later , all devices re-joined the network (I believe via the plug) but the osram plug and the previous connected devices are not visible in network map anymore. All of these devices working as usual but no way to see them on network map anymore.

[lolorc]

I repaired all my devices with a new pan_id. weird weird again, this is what I get:

And after I removed prise_01:

note that it still sees prise_01 despite it's been unplugged for more than 12h

And when I plug prise_01 again, prise_03 becomes router again for prise_0{1,2,4}. this is non sense.

[sukoshi1507]

Your outlet is certainly also a repeater. What model is it? Je suis intéressé.

Le ven. 31 août 2018 à 10:17, Laurent notifications@github.com a écrit :

I repaired all my devices with a new pan_id. weird weird again, this is what I get: [image: gr1] https://user-images.githubusercontent.com/10393822/44900774-c5d03b00-ad05-11e8-9e3c-1d8c69c61a82.png

And after I removed prise_01: [image: gr2] https://user-images.githubusercontent.com/10393822/44900788-d2ed2a00-ad05-11e8-8eaf-d00f4b81fb03.png

note that it still sees prise_01 despite it's been unplugged for more than 12h

And when I plug prise_01 again, prise_03 becomes router again for prise_0{1,2,4}. this is non sense.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Koenkk/zigbee2mqtt/issues/252#issuecomment-417588352, or mute the thread https://github.com/notifications/unsubscribe-auth/AQeB2pjYgI1gIpclHtiKD8VObCLwB9VUks5uWPEwgaJpZM4Vvz_F .

[lolorc]

all prise_* are OSRAM PLUG, model AB3257001NJ, yep they are Router. But either the network map is buggy, or they sucks at routing :-) why would prise_01 (2meters away from cc2531_2) be routed through prise_03 (2 floors upstairs)

zigbee2mqtt/prise_01 {"state":"OFF","linkquality":94} zigbee2mqtt/prise_02 {"state":"OFF","linkquality":26} zigbee2mqtt/prise_04 {"state":"OFF","linkquality":21} zigbee2mqtt/prise_03 {"state":"ON","linkquality":7}

[sukoshi1507]

2.4ghz spectrum can sometimes be very tricky in its propagation. I saw that numerous times in WiFi mesh network.

Merci pour la reference.

Le ven. 31 août 2018 à 10:42, Laurent notifications@github.com a écrit :

all prise_* are OSRAM PLUG, model AB3257001NJ, yep they are Router. But either the network map is buggy, or they sucks at routing :-) why would prise_01 (2meters away from cc2531_2) be routed through prise_03 (2 floors upstairs)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Koenkk/zigbee2mqtt/issues/252#issuecomment-417596180, or mute the thread https://github.com/notifications/unsubscribe-auth/AQeB2hqGzQKKc8Hrv4ZOHRSqK8mNaTwyks5uWPb-gaJpZM4Vvz_F .

[lolorc]

In this case I don't think it has anything to do with propag/reflection, a rpi with a 9db antenna can barely talk to my cisco AP 2 floors downstair, had to put another AP on the floor.

I'm going to try to use a zigbee sniffer. for cc2531 I might be able to use https://github.com/g-oikonomou/sensniff or https://github.com/andrewdodd/pyCCSniffer I also have a zigate (JN5169) and spare aqara devices (JN5168) I should also be able to use to sniff zigbee

[clockbrain]

Just wondering if the zigbee network is actually using multipath routing where it takes varying paths between nodes each time? Something to do with battery saving?

Has anyone tried taking a series of these network maps and overlaying them to see if there is a repeating pattern that represents the real mesh? Haven't done it myself yet as I am not using the dev branch.

See my comment earlier in the thread https://github.com/Koenkk/zigbee2mqtt/issues/252#issuecomment-410674657 where I map my zigbee2mqtt network with Digi XCTU. It shows multiple paths between nodes.

[splitice]

The lqiscan function assumes a parent Child relationship which is not true for routers. Routers can communicate with many other routers in a star fashion. The function is simplified for the ease of graph drawing. In my ZS fork lqiscan has been updated to be consistently breadth first too FYI.

On Fri., 31 Aug. 2018, 8:46 pm clockbrain, notifications@github.com wrote:

Just wondering if the zigbee network is actually using multipath routing where it takes varying paths between nodes each time? Something to do with battery saving?

Has anyone tried taking a series of these network maps and overlaying them to see if there is a repeating pattern that represents the real mesh? Haven't done it myself yet as I am not using the dev branch.

See my comment earlier in the thread #252 (comment) https://github.com/Koenkk/zigbee2mqtt/issues/252#issuecomment-410674657 where I map my zigbee2mqtt network with Digi XCTU. It shows multiple paths between nodes.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Koenkk/zigbee2mqtt/issues/252#issuecomment-417626816, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcmY2oWSjSIWn3JlPRAs_ieaRIJ7i8nks5uWRQGgaJpZM4Vvz_F .

[AnhDuc85]

is there any explanation for this case: xiaomi zigbee plug submits a link quality of 70 for about 10 hours. then I turn on the plug (remote), the link quality reduces to 25-30, there were no changes to my network during that time.

[clockbrain]

@AnhDuc85 any chance that whatever you have plugged into the zigbee plug is RF noisy? I had something similar with an old RF remote plug with connected floodlight that I could turn on remotely but had to turn off manually. The light was generating so much RF noise when on that it swamped the remote radio signal.

Try without anything plugged in or with a long lead to whatever you are switching.

[luci-11]

Hi guys, don't you think this informations are conflicting?

set 12 17:05:51 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:05:51 MQTT publish, topic: 'zigbee2mqtt/0x00124b0018e1cd14', payload: '{"state":false,"linkquality":202,"description":"0/0x00124B0018E1CD54","type":"COORD","rssi":1}'
set 12 17:05:51 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:05:51 MQTT publish, topic: 'zigbee2mqtt/0x00124b0018e1cd14', payload: '{"state":false,"linkquality":202,"description":"6144/0x00158D00022C8A75","type":"CHILD_RFD","rssi":170}'
set 12 17:06:21 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:21 Starting network scan...
set 12 17:06:22 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:22 Network scan completed
set 12 17:06:22 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:22 MQTT publish, topic: 'zigbee2mqtt/bridge/networkmap/graphviz', payload: 'digraph G {
set 12 17:06:22 raspberrypi npm[659]:   "0x00124b0018e1cd14" [label="0x00124b0018e1cd14 (online)"];
set 12 17:06:22 raspberrypi npm[659]:   "0x00124b0018e1cd14" -> "0x00124b0018e1cd54" [label="1"]
set 12 17:06:22 raspberrypi npm[659]:   "0x00158d00022c8a75" [label="0x00158d00022c8a75 (online)"];
set 12 17:06:22 raspberrypi npm[659]:   "0x00158d00022c8a75" -> "0x00124b0018e1cd54" [label="3"]
set 12 17:06:22 raspberrypi npm[659]:   "0x00158d000201a791" [label="0x00158d000201a791 (online)"];
set 12 17:06:22 raspberrypi npm[659]:   "0x00158d000201a791" -> "0x00124b0018e1cd54" [label="52"]
set 12 17:06:22 raspberrypi npm[659]: }'
set 12 17:06:51 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:51 MQTT publish, topic: 'zigbee2mqtt/0x00124b0018e1cd14', payload: '{"state":false,"linkquality":202,"description":"0/0x00124B0018E1CD54","type":"COORD","rssi":1}'
set 12 17:06:51 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:51 MQTT publish, topic: 'zigbee2mqtt/0x00124b0018e1cd14', payload: '{"state":false,"linkquality":202,"description":"6144/0x00158D00022C8A75","type":"CHILD_RFD","rssi":170}'
set 12 17:06:51 raspberrypi npm[659]:   zigbee2mqtt:info 2018-9-12 17:06:51 MQTT publish, topic: 'zigbee2mqtt/0x00124b0018e1cd14', payload: '{"state":false,"linkquality":202,"description":"6144/0x00158D00022C8A75","type":"CHILD_RFD","rssi":170}'

Please notice that in the map I get 3 devices connected to the coordinator (in reality I paired them to the COORD as still cannot join the network (pair devices) through router (this time I'm using CC2531 flashed as router ), but when reporting the topic reports the router id.
In fact the router is closer than the coordinator as they are the famous 2 sensors of issue #254 . Am I reading that log in a bad way?

Another strange thing, is that in HA the router state is "Disconnected" but this is another story I guess