search

dresden-elektronik / deconz-rest-plugin

deCONZ REST-API plugin to control ZigBee devices
BSD 3-Clause "New" or "Revised" License
1.88k stars 483 forks source link

groups/0 (all devices) not working #3744

Open popy2k14 opened 3 years ago

popy2k14 commented 3 years ago

Describe the bug

Yesterday i have updated to 2.05.88 from 02.05.84 and now my groups/0 doesnt work at all. I cant set it with phoscon "all off switch" or over rest api. There is a 404 error with HUEEssentials.

Steps to reproduce the behavior

Switch the group 0 as stated above.

Expected behavior

All lights should be controlable with group 0.

Screenshots

Environment

02.05.84 (downgraded again) 26660700

deCONZ Logs

Additional context

When ill do a get on group "0" manuall with the browser the answer is:

GET: http://192.168.0.9:8080/api/XXXXXXXXXX/groups/0 Answer: [{"error":{"address":"/groups/0","description":"resource, /groups/0, not available","type":3}}]

With other groups all is working.

ANy hints how i can get my group/0 working again?

MattWestb commented 3 years ago

Try with 0xFFF0 = 65520 that is the "new" groupe Al

popy2k14 commented 3 years ago

Thank you for your help. Sadly, not working on 2.05.84 after downgrade from 02.05.88. result is:

[{"error":{"address":"/groups/65520","description":"resource, /groups/65520, not available","type":3}}]

Any hints? Do i have to upgrade to 02.05.88 again to get it working?

PS.: why should the groups/0 be changed?

MattWestb commented 3 years ago

Then taking on look in zigbee.db in the table group wot number "All" is having but not the one that is marked "deleted"

MattWestb commented 3 years ago

About group 0 is in Zigbee 3 reserved for global scene commands and not as one trashcan for HA devices.

popy2k14 commented 3 years ago

ok, can you please guide me in the right direction regarding zigbee.db. Dont want to mess something up.

I just updated deconz yesterday to 02.05.88 and now my groups/0 is gone :-(

MattWestb commented 3 years ago

Sorry I was having wrong with the name but on my Pi its installed in /home/pi/.local/share/dresden-elektronik/deCONZ/zll.db deCONZ02

I have trying activating group 0 but deCONZ is marking deleted and automatic adding the 0xfff0 after restart for some month ago.

The 0xff09 is used by IKEAs 5 button remote scenes.

popy2k14 commented 3 years ago

I just have these ones... No group 0 deleted, but an 0xfff0 one named "All" which is marked as deleted.

groups

How can i get it back to a working state? just set it to normal?

Would it be accessible with groups/0 again over the rest api?

popy2k14 commented 3 years ago

Now when i'll do an:

http://192.168.0.9:8080/api/XXXXXXXXXX/groups/65520

i'll get:

{"action":{"alert":"none","bri":127,"colormode":"hs","ct":0,"effect":"none","hue":0,"on":false,"sat":127,"scene":null,"xy":[0,0]},"devicemembership":[],"etag":"xxxxxxxxxxxxxxxxxxxxxxxxxxx","id":"65520","lights":["3","2","1","4","7","6","5","8","9","10","11","12","13","14","16","15","19","18","17","20","21","25","22","23","24","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","42","43","44","45","46","48","49","50","51","52","53","41"],"name":"All","scenes":[],"state":{"all_on":false,"any_on":true},"type":"LightGroup"}

So it seems the all group is working under id 65520 now.

Sadly the "Power off" button in phoscon, HUEEssentials ... does not work because they will switch "0" instead of 65520.

Does it work when i just rename it to "0" in the sql table?

MattWestb commented 3 years ago

Its looks little normal deCONZ. You can trying changing "deteletd" to "normal" but I deCONZ is marking it deleted after next reboot. My installation have doing the 0xfff0 and giving it the name All. You can doing one new groupe "3" and adding all lights you like in it (its not adding new lights in it) or manual adding 0xfff0 and se wath deCONZ is doing on the next boot.

One warning only saving the DB then deCONZ is not running or it is overwriting you changes.

I think the system was little sleepy adding it after restart :-))

The last Q you can try but me experience is that deCONZ is redoing it to deleted after next boot.

MattWestb commented 3 years ago

Then you is having around 50 lights you should using real light groupe that is bounded to your remotes like the HUE dimmer switch so they is working if deCONZ is offline or having problems !!!

manup commented 3 years ago

Found the related commit which deletes group 0x0000 and adds a new non zero group which mostly happens to be 0xfff0: https://github.com/dresden-elektronik/deconz-rest-plugin/commit/590a8f1d1f363d98b8e452801b3983518ef91009

Devices are added to the new group 0xfff0.

The REST-API group "0" is mapped to Zigbee group 0xfff0.

I need to find the discussion behind this, not using (Zigbee) group 0x0000 has reasons but adding routers to that new 0xfff0 group has shown it's side effects and problems recently. I think it's best to use broadcast instead group cast when controlling REST-API group "0".

popy2k14 commented 3 years ago

@MattWestb: Thanks for all your help. Now i am back at 02.05.88.

I have now changed the 0xfff0 to 0x0000, stopped deconz before and than restarted. Group 0 was working for a short while. A few minutes later -> group 0 gone again (marked as deleted). And also 0xfff0 was added, marked as deleted. So now i have no zigbee "All" group at all.

@manup Thanks for stepping in. Is there a workaround how i can get group "0" to work again and not have to change it everywhere (fhem, HUE Essentials...) Maybe downgrade to 02.05.84, change the zll.db?

thx

manup commented 3 years ago

Actual group Zigbee group 0x0000 was never used, the REST-API group "0" was transformed into a broadcast to all routers, which I think is good, but likely there were other problems leading to the change.

Note that change was made over a year ago long before v2.5.84.

popy2k14 commented 3 years ago

thx, got you. So now i have done the following to get REST-API group "0" working again:

Until now (~5 Minutes) REST-API group "0" is working again.

Now its gone again :-( REST-API group "0" not working.

thx

popy2k14 commented 3 years ago

Now both are deleted. Seems deconz running an "job" which deletes my ALL groups. After starting deconz, the group is working (all lights are switching) for a few minutes until it returns:

[{"error":{"address":"/groups/0","description":"resource, /groups/0, not available","type":3}}]

After that the DB is looking like this:

image image

How can i get it to work again?

MattWestb commented 3 years ago

Great that Manup was not sleeping ;-))

In the latest Zigbee 3 papers is groupe 0 is reserved for scene command (globale one i think) and it was added then LL (light and occupancy) was added to ZCL. Its more zigbee groupes that is reserved at 0xffxx somthing and to 0xffff but i cant finding the paper its written for the moment (its about using touchlink in zigbee3 from NXP and sirlabs).

MattWestb commented 3 years ago

@popy2k14 If i reading the code change and the discussion is the groupe 0xfff0 in the database linked to the "group 0" in the API. The groupe 0 is inactivated and 0xfff0 added if not there after restart. From the discussion for the PR: Pore Philips HUE is not working with zigbee groupe 0 Little bad but OK in ZLL and ZB3) and OSRAM is not working with groups over 0xfffX (also OK in ZLL and ZB3) -> 0xfff0.

deCONZ can taking little time doing some magic after restart but if all is working the groupe 0 sould working in the API that hue essential is using (i dont like tapping on the switch in my handy then its being little dark all over then so i have not testing if its working).

My feeling is that the adding and deleting is working in the DB but perhaps not the linking for the API calls. "Some kind of magic (Queen)"

Then you is restarting deCONZ open the new debug window and you can see how the system is reading the groups from the DB and adding the lights in it.

popy2k14 commented 3 years ago

yeah i know, but the issue is that it "deletes" zigbee 0x0000 (which is ok because 0xfff0 is used) and 0xfff0 after a few minutes of deconz start.

So i can not use REST-API groups/0 (which should be internally 0xfff0).

How can i disable/prevent this behaviour?

MattWestb commented 3 years ago

If you having windows: Format C:

Sorry I think its one bug because its written in the discussion to the PR how it is working but perhaps is not tested in real then the DB part is working. I think @manup must taking one look on it.

As work around you can do one new normal groupe and adding all your (around 50) lights in it and rewriting your automations AND hope its works OK.

I was finding theise behaviour for 3 weeks ago then sniffing IKEAs remote and the scene select function that is not working. Was deleting all groups in one light and some seconds the 0xfff0 was automatically added by deCONZ all the time and i was hunting the groupe 0xff09. The DB i have seeing the deleting / cleaning the DB from crap after moving devices to ZHA.

MattWestb commented 3 years ago

@popy2k14 Pylips looks very interesting if having one Philips TV !!!

KodeCR commented 3 years ago

That's my commit. Sorry I broke things for you.

There is a config setting which stores in the database which zigbee groupid (normally 65520) is used for rest api group 0. This is in the database as under table "config2", variable "group0". If "group0" is 0, the zigbee group is deleted and a new one is created. Sounds like something is going wrong there. Could you find out what you have in the database for config2/group0?

popy2k14 commented 3 years ago

@KodeCR no problem :-) i can help debug It seems that your code is always triggered but also deleting 65520? Here is the db entry:

image

MattWestb commented 3 years ago

I dont have that proble then i dont using API (Only HUE Essentials) bit this os the record:

"14" "group0" "65520"

I think the DB part is working but not the linking to API.

popy2k14 commented 3 years ago

api linking is working! i can access REST-API "groups/0" after fixing the DB (set deleted to normal) and starting deconz. But after a few minutes REST-API "groups/0" isnt accessible again and DB flag is set to "deleted" on group 0x0000 and 0xfff0.

Currently fetching an debug log....

MattWestb commented 3 years ago

In HA i have the IKEA scene defalt groupe "Group 65289" from deCONZ but no 0 or 0xfff0 perhaps is hidden or the linking is not working to API.

KodeCR commented 3 years ago

Thanks, I can't see in the commit I made how it would be deleted. In the original commit that only happened if group0 from the database was 0.

I did commit a "fix", but that one deleted the group only if the zigbee address is 0.

So it's working right after starting deCONZ?

popy2k14 commented 3 years ago

After "fixing" DB (set deleted to normal) on 0xfff0 entry and starting deconz it works for a few minutes. Should ill set the config2/group0 entry to "0" and the All group to 0x0000 and try again?

popy2k14 commented 3 years ago

i think i found the issue, see here from the debug log:

22:01:57:017 [INFO] - No button handler for: FYRTUR block-out roller blind endpoint: 0x01 cluster: BASIC (0x0000) command: 0x0A payload[0]: 000 22:01:57:017 ZCL attribute report 0x680AE2FFFE36D57F for cluster: 0x0000, ep: 0x01, frame control: 0x08, mfcode: 0x0000 22:01:57:017 payload: 00404207322e322e303039 22:01:57:031 Node data 0x680ae2fffe36d57f profileId: 0x0104, clusterId: 0x0000 22:01:57:032 Update Sensor 0x680AE2FFFE36D57F Basic Cluster 22:01:57:032 0x680AE2FFFE36D57F: update ZCL value 0x01/0x0000/0x4000 after 0 s 22:01:57:032 [INFO] - No button handler for: FYRTUR block-out roller blind endpoint: 0x01 cluster: BASIC (0x0000) command: 0x0A payload[0]: 000 22:01:57:032 ZCL attribute report 0x680AE2FFFE36D57F for cluster: 0x0000, ep: 0x01, frame control: 0x08, mfcode: 0x0000 22:01:57:032 payload: 00404207322e322e303039 22:01:57:034 Websocket 192.168.0.1:57850 send message: {"config":{"group":"65520","on":true,"reachable":true},"e":"changed","id":"77","r":"sensors","t":"event","uniqueid":"68:0a:e2:ff:fe:36:d5:7f-01-0001"} (ret = 32410024) 22:01:57:035 Websocket 192.168.0.9:33928 send message: {"config":{"group":"65520","on":true,"reachable":true},"e":"changed","id":"77","r":"sensors","t":"event","uniqueid":"68:0a:e2:ff:fe:36:d5:7f-01-0001"} (ret = 32410024) 22:01:57:037 attach group 65520 to sensor 77 22:01:57:077 Node data 0x680ae2fffe36d57f profileId: 0x0104, clusterId: 0x0000 22:01:57:077 Update Sensor 0x680AE2FFFE36D57F Basic Cluster 22:01:57:078 0x680AE2FFFE36D57F: update ZCL value 0x01/0x0000/0x4000 after 0 s 22:01:57:078 [INFO] - No button handler for: FYRTUR block-out roller blind endpoint: 0x01 cluster: BASIC (0x0000) command: 0x0A payload[0]: 000 22:01:57:078 ZCL attribute report 0x680AE2FFFE36D57F for cluster: 0x0000, ep: 0x01, frame control: 0x08, mfcode: 0x0000 22:01:57:078 payload: 00404207322e322e303039 22:01:57:080 Websocket 192.168.0.1:57850 send message: {"config":{"group":"14","on":true,"reachable":true},"e":"changed","id":"77","r":"sensors","t":"event","uniqueid":"68:0a:e2:ff:fe:36:d5:7f-01-0001"} (ret = 32410024) 22:01:57:080 Websocket 192.168.0.9:33928 send message: {"config":{"group":"14","on":true,"reachable":true},"e":"changed","id":"77","r":"sensors","t":"event","uniqueid":"68:0a:e2:ff:fe:36:d5:7f-01-0001"} (ret = 32410024) 22:01:57:082 delete old group 65520 of sensor Rollo Julian 22:01:57:082 attach group 14 to sensor 77

The interesting part is:

22:01:57:082 delete old group 65520 of sensor Rollo Julian

Thats an ikea fyrtur rollo and deconz deletes the 0xfff0 (65520) group here.

Any hints why?

KodeCR commented 3 years ago

Having config2/group0 set to 65520, and having the 0xfff0 group entry as "normal" is the default behaviour and how it is currently working for me. I need to figure out why the group is deleted after a while. Perhaps the debug log gives some information. I'm afraid I'll have to look at it further tomorrow though.

popy2k14 commented 3 years ago

I think its not your code causing the deletetion of the group 0xfff0. Please see above. How can i prevent deconz from deleting the 0xfff0, is there any other table/assignment where i can delete the 0xfff0 from sensor 77 which is "Rollo Julian"?

KodeCR commented 3 years ago

Ah, thanks, that is interesting. I think that is what it does when the group is deleted in the rest api, and it then removes that group from the group tables in each light. Now this is a sensor, so that might be a clue. Needs some digging.

popy2k14 commented 3 years ago

Also thanks for your support. I got a little further. When i am doing an "Get Group Membership response" in the GUI i'll get the 0xfff0 from "Rollo Julian":

image

I am not so deep in the GUI, how can i delete this response which i think leads to the issue.

thx

MattWestb commented 3 years ago

@popy2k14 I think deCONZ was adding all lights and sensors to groupe 0 before for getting the reporting working to the coordinator (most zigbee system is doing that). ZHA is calling groupe 0 coordinator groupe. And then changing to 0xfff0 it also adding all devices so the reporting from the devices is working to the coordinator (I think deCONZ can listening for reports on all groups that most other NCP ca not doing) so i think Julians Rollo is OK in the 0xfff0 groupe.

MattWestb commented 3 years ago

If using the command in the group cluster "delete all groups" but im 110% deCONZ is adding it back in some seconds (my experience with IKEA GU10 WS bulbs).

MattWestb commented 3 years ago

sorry its under your screen shutt and its "remove all groups"

If you is having IKEA bulbs and remote you shoud adding the groupe 0xff09 in the bulbs then you can using the scene control of the remote :-))

MattWestb commented 3 years ago

The Julian Rollo have 2 groupes the first is the normal you have adding for stealing it and the second is deCONZ added 0xfff0 and if your system is old also 0x0000.

popy2k14 commented 3 years ago

THanks for all the info, it seems you having much much more knowledge that i have regarding zigbee networks and ikea devices :-))

I have now just use "remove group" and entered the 0xfff0 into it. After another request it showed me just one entry with the correct group id 0x000e (ikea switch and blind device).

Now i have shut down the GUI and started the minimal headless version (which i normally use) and until now REST-API group "0" is working since 7 minutes...

...before pressing "Comment" group "0" is gone again :-((

Dont know whats going on and how i can solve it. So you are right, my try to solve it does'nt help. Cant figure out why the 65520 is delete from deconz.

Any hints?

MattWestb commented 3 years ago

Its one undocumented future (bug) for shore !!

And im also 110% shore that deCONZ is adding the 0xfff0 to Julians Rollo then its starting and also after some seconds after you have deleting it in the rollo.

MattWestb commented 3 years ago

The 0xfff0 / 65520 groupe is also not added i HA as normal groupe that all other groups ar but it can being flagged not being added in HA (and by mistake also in API).

popy2k14 commented 3 years ago

Yeah you are right. I have now repeatedly requested group memberships after starting deconz-gui. first it's one group with id 0x000e (which is the correct group for the switch) and after a few minutes 0xfff0 is added by deconz.

One thing i have noticed is that all other 6x ikea blinds i own have "3" group memberships and "Rollo Julian" just "2". Dont know if thats the issue!?

Tomorrow i will try to reset the blind, because now childrens are sleeping in the room :-))

MattWestb commented 3 years ago

If deCONZ have making it right is the remote bonded to the groupe so its working then deCONZ is offline. If you is using 2 remotes for the blinds they is bounded to the same group. If you have more blinds and one controller both blinds is the the same group as the controller. If you like Juliand rollos being controlled with the other 6 then add groupe 3 to it and its controlled from both groupe 2 and 3 controller.

Its no magic its only Zigbee lights ;-))

Edit: I was miss reading you !!!

I you have the rollos for around one year they is having the old groupe 0 = 0x0000. the new 0xfff0 and your controller groupe => 3 If you have resetting the device its gon and deCONZ have adding the new "groupe 0" = 0xfff0 and your controls groupe. => 2

popy2k14 commented 3 years ago

no, you got me wrong. I mean't the "group memberships" in the GUI not how many remotes is controlling the blind. For "Rollo Julian" just one remote is added to trhe same group.

MattWestb commented 3 years ago

The pre post eded a bow !

And in the GUI you can only see the last (highest number) groupe then reading it from the device :-( Then sniffing the device is sending the complete list with all groupe ids.

popy2k14 commented 3 years ago

Tried also to change the config/group0 entry to 65519 and groups entry for all to 0xFFEF without success. REST-API groups/0 is working for about ~7 minutes and gets than deleted from deconz.

Will continue tomorrow, maybe @manup can take a look at the log above and can answer why my deconz setup is deleting the "All" group, no matter which id it has.

Thank you

MattWestb commented 3 years ago

@manup The light group is is special for LL in zigbee 3 and is not moderate for HA. But as deCONZ is supporting both profiles its best following the LL group allocating for lights in ZB3. Have finding the reference to the 0x0000 groupe Silab UG103.9: ZLL Fundamentals 4.3 Addressing. I have seen the same from NXP but cant finding it and have not looking thru the alliance docks.

popy2k14 commented 3 years ago

@manup can i send you the complete debug log? I was'nt able to solve the issue that, regardless which group i am activating as group0, it get "deleted".

popy2k14 commented 3 years ago

Next try: removed the battery from "Rollo Julian", fixed DB (set 0xfff0 to normal) restarted deconz. REST-API groups/0 is working for ~5 minutes -> then it was set as deleted again in the DB :-(

Any hints i can stop deconz from removing 0xfff0 = REST-API groups/0 ?

thx

popy2k14 commented 3 years ago

I have made 3x debug logs which i can provide if needed (dont want to upload it anywhere because of sensible data of my network in it). Between the logs i have disconnected power to several blinds -> did'nt help. In every log is the line "delete old group 65520 of sensor ..." (search from the bottom). I think it's an bug which gets triggered in my setup.

Anyway, as a workaround and suggested, i have made an new group named "All" and put all lights in it. That works frrom fhem & HUEEssentials and does'nt get deleted.

Would be nice if this can be fixed in a future update. thx

MattWestb commented 3 years ago

Good work done with debugging !! I hope the devs gett time to digging and finding the bug. I have not trying the all grupe then its the hole apartment getting dark. But the 0xfff0 groupe is not marked deleted in the database.

KodeCR commented 3 years ago

Have finding the reference to the 0x0000 groupe Silab UG103.9: ZLL Fundamentals 4.3 Addressing.

Thanks for this, I wasn't aware of this, in particular:

The group identifiers need to be unique in the network and their range is (0x0001…0xFEFF). Group identifier 0x0000 is reserved
for the default group in the ZCL scene cluster, and group identifiers in the range (0xFF00…0xFFFF) are reserved.

I'll update the code taking that into account.

It might be the cause of the issue, but I doubt it.

deCONZ periodically synchronised the group membership of all nodes with the way it is in the REST api. The message "delete old group 65520 of sensor Rollo Julian" is definitely part of this, it checks if the node is part of the group in the rest api, if not it considers the group orphaned and removes it. Not quite sure why it then completely removes the group instead of just removing that group from the node.... But... I think I found the bug, in that the code has a condition:

i->address() != 0

which I believe should be:

i->address() != gwGroup0

I'll carefully go through it and create a PR (might take a few days), if you know how to compile the plugin, perhaps you could check if that works.