Documentation and support for non or partially recognized devices

Utopiah commented 3 years ago

There is a page dedicated to documentation about adding a new device Adding-a-new-device.md but despite the technical competency of all people raising issues (most if not all added logs and were able to create an issue here on Github thus fairly safe to assume they have some knowledge about software development) remains unsolved but, more worryingly, unanswered.

One of they key advantage of WebThingsIO that convinced to use the platform compared to alternatives was about being about source and rely on standards like Zigbee. I thought that it would bring safety and that unless a firmware was flawed there would always be a way. Unfortunately in practice I'm afraid those issues without replied are a symptom of a problematic situation. There is no expectation for the issues to be solved at all but that there is no pointer provided on how feels like a key component of the ecosystem, Things support, is dysfunctional. I can't imagine this be due to gatekeeping from anybody in the project yet, in practice, it seems functionally equivalent. I would rather not speculate on the reasons behind this situation.

Could anyone please explain the process to follow in order to provide supports for devices that are not working at all or as expected?

mrstegeman commented 3 years ago

The main problem is that I'm the de facto maintainer of this adapter, but:

I've been spending most of my time working on the gateway.
I've only ever done minimal development on this add-on, and it's quite large and complex.

@davehylands (@dhylands? not sure which you're using these days) is the original creator of this add-on, but I know he doesn't have much time to devote to it these days. @chas-iot, @frederic34, and @mrsteakhouse have done a fair amount of work lately, so they may be able to offer some better mentorship.

With all that said, pull requests are always welcome. I promise you're not being ignored, as I see all of the issues coming in. I just don't have time to work on them.

rzr commented 3 years ago

Would a wiki page permits users to help each others ? there are also the webthing forum but I think wikis are better in longer term ...

mrstegeman commented 3 years ago

There are already several resources here: https://github.com/WebThingsIO/wiki/wiki#zigbee

It’s publicly editable, so anybody is free to add more content.

flatsiedatsie commented 3 years ago

Coïncidentally, there would be a way to offload zigbee support to the community, as I just noticed this PR for the unofficial Zigbee2MQTT addon.

https://github.com/kabbi/zigbee2mqtt-adapter/pull/25

Perhaps if that addon would be made official (I'm a co-maintained now) it would allow the Zigbee2MQTT community to do some heavy lifting. It could give users options?

mrstegeman commented 3 years ago

I was open to this a while back and still am. I was just hoping it could be made more generic in some way and that somebody was going to actively maintain it.

Utopiah commented 3 years ago

@mrstegeman to clarify I'm not criticizing the pace of accepting PR and I completely understand that you don't have the time. The goal of this issue wasn't to suggest more work from maintainer but precisely the opposite. How can we find a way not to be dependent on few people when the core of the project, I believe, is perfectly functional and thus does not require the same attention anymore?

Regarding architecture choices, with e.g Zigbee2MQTT, I don't have the expertise to share a meaningful opinion. That being said, the JSON snippet looks precisely like an example newcomer could extend upon.

Overall the wiki links are useful but they don't seem to directly address the recurring question of "How can I add support for my device?" to newcomers. Again I don't have the knowledge for that since I've never done it, so I'm naive and maybe optimistic about it. Maybe it requires understanding of the Zigbee protocol, dedicated hardware or reverse engineering that make it unpractical for newcomers or only in 0.0001% of situations. In that case the documentation won't necessarily have a very big impact. If though it's "as easy" as changing the type of a device in a JSON file for 10% of cases then I'd argue it's worthwhile.

mrstegeman commented 3 years ago

In some cases, it's fairly straightforward, i.e. with the Xiaomi devices. They have their own file, where things are laid out in a nice, understandable way. In most other cases, things filter through a bunch of logic in zb-classifier.js, which can be a bit hard to follow. Most times, it's just a matter of following the logic while looking at your JSON file and seeing where things are breaking down.

chas-iot commented 3 years ago

I'd like to do more development of this adapter. However the issue is not just about maintainers and developers. It's also about testers.

I have patches awaiting testing. I am not comfortable to go ahead and ask @mrstegeman to apply them anyway, without independent confirmation that the patches work in different configurations.

And what this means is that when I develop further, I'm not testing in a fully 'normal' environment, so all of my subsequent work deserves additional inspection and testing. The patches mentioned above are necessary to make my slightly odd configuration work. So there's a bit of a chicken and egg situation for me.

flatsiedatsie commented 3 years ago

Could the Zigbee2MQTT adapter's proposed approach be possible/useful here?

If a device is in it's list of supported devices, it uses that to create an optimal thing.
If it's not, it does a generic translation based on variable types, and creates a thing that way.

Utopiah commented 3 years ago

An example of naive newcomers I modified the .webthings/data/zigbee-adapter/zb-XXXX.json only to notice that it doesn't get loaded and get over-written. Obviously the error was mine but it shows the confusion from newcomers about the flow of the Gateway. How Things get detected and not just how but also where they can be modified then if they work locally how to share back the result as a PR.

davehylands commented 3 years ago

The zb-XXX.json file acts as a cache. The issue is that once a battery powered device has been paired, it may only communicate with the gateway once every few hours. When the zigbee adapter is loaded, it reads the zb-XXX.json file and updates it with new information that it receives while running. Most of the information is obtained when pairing a new device, but additional information sometimes trickles in later (like state changes).

Editing the zb-XXX.json file is ok, as long as the zigbee adapter is currently unloaded. Otherwise it will overwrite the file anytime it receives new information from any of the devices.

flatsiedatsie commented 3 years ago

The Zigbee2MQTT adapter is now available.

I'd love to hear any user experiences, but please share them at https://github.com/kabbi/zigbee2mqtt-adapter/issues

tim-hellhake commented 3 years ago

Coïncidentally, there would be a way to offload zigbee support to the community

I think that's a fairly strong argument for using zigbee2mqtt. There are a lot of issues in this repo about devices zigbee2mqtt already supports. I see no point in spending time on that if the community already solved it in a platform-agnostic way.

tim-hellhake commented 3 years ago

I'd love to hear any user experiences, but please share them at https://github.com/kabbi/zigbee2mqtt-adapter/issues

I had trouble adding my Xiaomi Wireless Switch with the zigbee-adapter. With the zigbee2mqtt addon it worked right away. So, great work @kabbi @flatsiedatsie! I already thought about returning the switch.

kgiori commented 3 years ago

@tim-hellhake - what Xiaomi wireless switch do you have? And is there any problem running the regular zigbee add-on in parallel with the zigbee2mqtt one?

mrsteakhouse commented 3 years ago

Both the zigbee2mqtt-addon and the zigbee-addon will need to open the serial port to the coordinator hardware in order to receive and transmit messages. Since linux does not support opening one serial port to multiple applications, both addons won't be able to function simultaneously. You would need two independent coordinators and networks.

I switched during the last days to zigbee2mqtt, because i wanted a layer of abstraction between device communication and control. Also things run now more smoothly for me.

tim-hellhake commented 3 years ago

@kgiori This one: https://www.zigbee2mqtt.io/devices/WXKG01LM.html

kgiori commented 3 years ago

This one: https://www.zigbee2mqtt.io/devices/WXKG01LM.html I have that one too and it (sort of) works for me with the regular zigbee add-on -- it's just really odd, so I have to do weird rule contortions to make it useful. Would the same properties (multiclick number and pressed boolean) result from using the zigbee2mqtt add-on? Can the zigbee2mqtt add-on support all the same devices as the regular zigbee add-on?

flatsiedatsie commented 3 years ago

In theory you could just try it? Disable the zigbee addon and then enable the zigbee2mqtt addon. Then pair the device. If you're not happy with it, just switch back.

mrsteakhouse commented 3 years ago

In thory you could just try it? Disable the zigbee addon and then enable the zigbee2mqtt addon. Then pair the device. If you're not happy with it, just switch back.

Those steps must be taken with caution. Parameters like the Pan-ID and the network key will be regenerated when using zigbee2mqtt resulting in the need of re-pairing all zigbee devices. However, switching back to the zigbee-addon might be less invasive.

flatsiedatsie commented 3 years ago

Yikes, forget I said that :-D

Utopiah commented 3 years ago

Just gave it a go after another Thing was connectable but with the wrong type and thus unusable. Both got correctly recognized by the Zigbee2MQTT adapter.

I'm very happy about it but, naive question, doesn't it shift away the problem to another adapter? Should we expect every Zigbee device we buy to be supported by Zigbee2MQTT?

tim-hellhake commented 3 years ago

@kgiori I had the same problems as you. The new update of the zigbee-adapter also supports supports zigbee2mqtt. You need to start your own zigbee2mqtt instance though. With the zigbee-adapter my rule looks like this:

tim-hellhake commented 3 years ago

@Utopiah

With the zigbee-adapter and zigbee2mqtt it looks about this for me:

Should we expect every Zigbee device we buy to be supported by Zigbee2MQTT?

Unless someone has the time to actively maintain the current implementation of the Zigbee stack I see no other option. I may be wrong but I think it's also far easier to add new devices to Zigbee2MQTT.

Utopiah commented 3 years ago

it's also far easier to add new devices to Zigbee2MQTT.

which links back the original issue, how would a complete newcomer to WebThings (let's say me) add their partly or non supported thing? Any link to an example with the debugging process for a new Things for Zigbee2MQTT?

PS: this morning I switched my small setup (~12 Things) to Zigbee2MQTT, very happy with the result, so far everything works.

tim-hellhake commented 3 years ago

Any link to an example with the debugging process for a new Things for Zigbee2MQTT?

The is an excellent howto.

WebThingsIO / zigbee-adapter

Documentation and support for non or partially recognized devices #278