Versions after 2.05.85 Sensor State Changes Lost to Home Assistant

majorsl commented 3 years ago

Describe the bug

I use deCONZ with the Home Assistant integration: https://www.home-assistant.io/integrations/deconz/

Upgrading to the beta 2.6.0 of deCONZ stops all my sensor states from changing in Home Assistant, but the states do still show in Phoscon.

I tried removing/re-adding the integration with no change. Of note, when I re-added the integration I had a device shown by HA that is "null" just () with no data or information. This does not appear in Phoscon or deCONZ. Erroneous data being sent by the API?

Steps to reproduce the behavior

If the problem is reproducable, list the steps here:

Use Home Assistant
Install a version after 2.05.85
Watch your sensors no longer work in Home Assistant

Expected behavior

Sensors toggle states in Home Assistant.

Screenshots

Environment

Host system: Raspberry
Running method: Marthoc Docker container
Firmware version: (26660700)
deCONZ version: > greater than 2.05.85
Device: ConBee II
Do you use an USB extension cable: yes
Is there any other USB or serial devices connected to the host system? no

deCONZ Logs

Additional context

Has been working perfectly with Home Assistant up to this point. I have reverted away from the beta, but will be happy to try again if anything is found or a fix is available.

Kane610 commented 3 years ago

It would be good if you could share debug logs from hass. Enable debug per deconz integration instructions and share them here

Kane610 commented 3 years ago

Typically when no state changes are observed it's the websocket not setting up properly

majorsl commented 3 years ago

We have a thread going on reddit, someone did few tests and it seems the last version that is working is 2.05.84 (I think that is 85). I'm about to revert to that if I can figure out how to use a local image I still have - never done that. https://www.reddit.com/r/homeassistant/comments/jx7lxo/0118_and_deconz_phoscon_issues/

majorsl commented 3 years ago

Confirmed: 2.05.85 is the last version that worked. Stepping back through the Docker container versions one by one to it finally started sending sensor states to Home Assistant again.

Mimiix commented 3 years ago

I have no issues on 2.60.0 with HA and deCONZ. Running deCONZ in a seperate VM.

manup commented 3 years ago

I've looked in the GIT diff between v2.5.85 and v2.5.86, the only REST-API difference here seems to be:

Which is related to ELKO Super TR:

{
  "config": {
       "temperaturemeasurement" : "air sensor"
  }
}

Could it be that?

SwoopX commented 3 years ago

@manup Already discussed with @Kane610 . That's why I updated the API documentation and added when something new was introduced. Cannot be sure however, that I caught everything.

I guess a bit more was introduced. We also have one issue where button_maps.json appears to be missing.

manup commented 3 years ago

According to the diff there seems to only the above and one error message which is different in the REST-API output.

Otherwise I can't see anything :thinking:

SwoopX commented 3 years ago

A bit more. Besides the config part already mentioned, also this as state (both applicable for thermostats only) grafik

https://github.com/dresden-elektronik/deconz-rest-plugin/pull/3329

manup commented 3 years ago

So the question is if these devices are paired in the problematic installations?

manup commented 3 years ago

The PR seems to only add new attributes but doesn't change existing REST-API attributes. This hopefully shouldn't break a client.

Kane610 commented 3 years ago

The deconz integration creates standard sensors if there isn't added explicit support for the sensor so it shouldn't break but it might not expose the best information about it

indomus commented 3 years ago

Same problem here. https://github.com/dresden-elektronik/deconz-rest-plugin/issues/3645

majorsl commented 3 years ago

Same problem here. #3645

My goal yesterday was to get up and running as quickly as possible. What can I provide that hasn't been discussed in the above issue to try to help?

I'm using Docker so it isn't too much of an issue to create latest or stable to get some information and then I can switch back to a working environment.

Fortunately, I don't have many devices and no repeaters yet, so if it is a device causing it maybe changes there can be a clue. I have these: SNZB-04 (6), Xiaomi Aqara Temp/Humid Sensor (1), CentraLite 3323-G (3)

adrha commented 3 years ago

Hey @manup first of all thank you for your great support!

I'm having the same issues as others here in this thread but recently figured out that the .85-version is the first bugy-release (and not 86 as reported earlier). For me a roll-back to 84 fixed it. The interesting point is, that others in this thread are using Aqara Temp/Humidity sensors, which i also use and they are having issues as well.

Florian-Schmidt commented 3 years ago

Same issue here: Suddenly HA automations stopped working during the last days.

Tried to update deconz and ConBee II to: deconz: 2.06.00 firmware: 26670700

Every device was not reachable, over the day some devices came back. Now downgraded back to. deconz: 2.05.86 firmware: 26660700

Still the same, updated deconz again to 2.06.00 but no changes. Also tried an older version 2.05.73 but still the same devices available.

Unfortunately I don't have a 2.05.84 available.

Florian-Schmidt commented 3 years ago

Ok, build marthoc/deconz container locally and now running.

deconz: 2.05.84 firmware: 26650700

But still the same, there are some connections shown in VNC viewer but many devices are missing and the zigbee network doesn't build up.

Florian-Schmidt commented 3 years ago

Ok, now the first lights are coming back again and zigbee network grows up slowly...

Anyone an idea how to speed up the reconnect of known devices?

SwoopX commented 3 years ago

@Florian-Schmidt Please do not hijack other issues. You mentioned that none of your devices are reachable, which is a completely different topic so for this, please raise your own issue.

To answer your question: You can press 0 (zero) in deconz GUI while selecting a routing node, or F5.

Florian-Schmidt commented 3 years ago

@SwoopX Thanks for you answer. Sorry, thought this could be helpful as these releases seem to have several similar issues...

Actually Zigee network seems to continue growing up more and more. Going to wait some more days and hope to reach complete network again. In case of getting disappointed by incomplete network, I'll open new ticket.

stale[bot] commented 3 years ago

As there hasn't been any response in 21 days, this issue has been automatically marked as stale. At OP: Please either close this issue or keep it active It will be closed in 7 days if no further activity occurs.

majorsl commented 3 years ago

As there hasn't been any response in 21 days, this issue has been automatically marked as stale. At OP: Please either close this issue or keep it active It will be closed in 7 days if no further activity occurs.

Still an issue.

adrha commented 3 years ago

Has someone got it to work from you guys? I'm still running on .84 version. The latest version does still not work with HomeAssistant. Is there a way to export the device-config and import it on the new setup without repairing all nodes?

majorsl commented 3 years ago

Has someone got it to work from you guys? I'm still running on .84 version. The latest version does still not work with HomeAssistant. Is there a way to export the device-config and import it on the new setup without repairing all nodes?

Not that I'm aware. I was going to try it again today based on #3977 which seems similar. I had not waited hours for the sensors to come back so i thought I'd give that a go.

I agree - rebuilding from scratch just over an upgrade isn't optimal, but the later builds have some device support I want to use. :(

thecem commented 3 years ago

Which HA Version 2020.12.1? Tried Phoscon Version 2.07.01? Tried latest Conbee FW II 26680700?

majorsl commented 3 years ago

Which HA Version 2020.12.1? Tried Phoscon Version 2.07.01? Tried latest Conbee FW II 26680700?

Yes. Already was running HA 2020.12.1. I just updated to the latest stable version and upgraded my firmware to 26680700.

Again, the same issue where the sensors trigger in deCONZ and Phoscon, but do not register in Home Assistant. Lights work okay.

As mentioned in #3977, I am going to wait 6-8 hours and see if they eventually start to work again. If not, I will revert to the last known working deCONZ.

thecem commented 3 years ago

did you try the services in HA:

deconz.device_refresh

or

deconz.remove_orphaned_entries

majorsl commented 3 years ago

did you try the services in HA:

deconz.device_refresh

or

deconz.remove_orphaned_entries

I just tried both, still no status updates in Home Assistant.

Mimiix commented 3 years ago

@Kane610 can you pitch in and see if this is something on our end?

Kane610 commented 3 years ago

Is there changes to how the docker is configured because this typically means that the websocket isn't set up correctly

majorsl commented 3 years ago

Is there changes to how the docker is configured because this typically means that the websocket isn't set up correctly

On my end, no. I am using Portainer to work with deCONZ. I shut down 2.05.85, did a Duplicate/Edit of the container and changed it to marthoc/deconz:stable and started it. No changes whatsoever other than the latest stable version.

I believe the web socket is working because I can control (for example) an outlet with Home Assistant just fine. It is just sensors that are not working which makes my issue identical to #3977.

Kane610 commented 3 years ago

Sending command and receiving updates are two different things. Does state updates get reflected to outlets and lights? Enable debug logs in hass for deconz integration. Do you see any events over time?

majorsl commented 3 years ago

Enable debug logs in hass for deconz integration.

I don't know how to do this and I don't see an option in the GUI for that.

majorsl commented 3 years ago

Also, I don't know if it means the same thing, if I toggle a light on/off in Phoscon I do see the state change in Home Assistant GUI. So HA is receiving what I call an "update" to those devices.

Kane610 commented 3 years ago

Enable debug logs in hass for deconz integration.

I don't know how to do this and I don't see an option in the GUI for that.

Check deconz integration documentation for how to enable debug

adrha commented 3 years ago

In my setup i use some Aqara Opple-switches (8CH). After updating to latest or stable, the events of button-presses are not forwarded to homeassistant anymore. Also temperature/humidity-sensors-updates are not forwarded anymore. When querying through GET-Request on the API the updated values of the sensors are returned.

So obviously just the websocket-part is faulty.

As far as i know, there is an option to manually download the deconz-config in the frontend. After uploading this to a fresh installation, do i have go re-pair all devices or should it work directly?

majorsl commented 3 years ago

In my setup i use some Aqara Opple-switches (8CH). After updating to latest or stable, the events of button-presses are not forwarded to homeassistant anymore. Also temperature/humidity-sensors-updates are not forwarded anymore. When querying through GET-Request on the API the updated values of the sensors are returned.

So obviously just the websocket-part is faulty.

As far as i know, there is an option to manually download the deconz-config in the frontend. After uploading this to a fresh installation, do i have go re-pair all devices or should it work directly?

If you try it and it works, let me know. I too have a temp/humidity sensor and it is also not working with Home Assistant. Something clearly has changed/broken in these later versions with many people reporting the same thing in several issues here and via replies to them.

majorsl commented 3 years ago

I enabled the debug log in HA. This is what I see when I use Phoscon to toggle the plug near the couch "on".

2020-12-28 12:42:44 DEBUG (MainThread) [pydeconz.websocket] {"e":"changed","id":"2","r":"lights","state":{"alert":null,"on":true,"reachable":true},"t":"event","uniqueid":"00:12:4b:00:1e:72:03:06-01"} 2020-12-28 12:42:44 DEBUG (MainThread) [pydeconz.websocket] {"e":"changed","id":"65520","r":"groups","state":{"all_on":true,"any_on":true},"t":"event"} 2020-12-28 12:42:44 DEBUG (MainThread) [pydeconz.websocket] {"e":"changed","id":"1","r":"groups","state":{"all_on":true,"any_on":true},"t":"event"} 2020-12-28 12:42:45 DEBUG (MainThread) [pydeconz.websocket] {"attr":{"id":"2","lastannounced":null,"lastseen":"2020-12-28T17:42Z","manufacturername":"SONOFF","modelid":"S31 Lite zb","name":"Couch","swversion":null,"type":"On/Off plug-in unit","uniqueid":"00:12:4b:00:1e:72:03:06-01"},"e":"changed","id":"2","r":"lights","t":"event","uniqueid":"00:12:4b:00:1e:72:03:06-01"}

I then tripped two door/window sensors. Their state changed in Phoscon, but not in Home Assistant. There were no logs generated for the sensor events.

Kane610 commented 3 years ago

@Mimiix some events reach hass but not others so it sounds like something on deconz side

majorsl commented 3 years ago

@Kane610 thank you for helping me get in the right direction for the logs.

Mimiix commented 3 years ago

@majorsl can you provide some logs?

To enable logging. Open deCONZ and click Help. After that, click Debug View. The following debug levels need to be enabled for proper logging: INFO, ERROR,ERROR_L2,APS,APS_L2.

s0ftice commented 3 years ago

@Mimiix I provided logs a week ago here https://github.com/dresden-elektronik/deconz-rest-plugin/issues/3977

Mimiix commented 3 years ago

Please don't hijack others issues...

majorsl commented 3 years ago

@Mimiix we do have same issue, perhaps his will help. After enabling, do you want me to do the same tests I did for Mimiix? Toggle a light/plug then the sensors? Thanks for helping.

s0ftice commented 3 years ago

We believe it's the same

Mimiix commented 3 years ago

@Mimiix we do have same issue, perhaps his will help. After enabling, do you want me to do the same tests I did for Mimiix? Toggle a light/plug then the sensors? Thanks for helping.

Yes.

Your symptoms are the same, but we can't be sure yet. What else do your setups share other than the same problem?

For example: I have ikea sensors and xiaomi sensors working and I have no issues at all. If it is that widespread, more should be affected.

majorsl commented 3 years ago

@Mimiix I'm running this in Docker. I can get into debug and see the logging, but I can't save it or copy/paste out of VNC to get them here. I can probably screenshot the sections and past in numbered order here, unless you have a better option...

majorsl commented 3 years ago

@Mimiix & @Kane610 ...and now everything is working all at once again, just like @s0ftice said in his issue. I did nothing.

About 5 hours since I started up deCONZ. Interesting that in my Time Zone I'm UTC-05:00.

Mimiix commented 3 years ago

@Mimiix I'm running this in Docker. I can get into debug and see the logging, but I can't save it or copy/paste out of VNC to get them here. I can probably screenshot the sections and past in numbered order here, unless you have a better option...

Use vnc to connect to it?

s0ftice commented 3 years ago

I propose to mark mine (https://github.com/dresden-elektronik/deconz-rest-plugin/issues/3977) as duplicate of this issue. As @majorsl observed the same magic "self-healing" after a few hours.

@Mimiix agree, this does not seem to be widespread and unfortunately, I don't see what our setups have in common which would easily explain what causes the issue. My setup is as vanilla as possible (see environment of 3977).

dresden-elektronik / deconz-rest-plugin