Can't force remove old CC253X Routers

[dortamur]

Bug Report

What happened

Some time ago, I physically removed two CC2531 (using CC2530.ROUTER firmware) routers and repurposed them, without cleanly removing them. Now, when I try to forget / force-remove them, I get the error "device_force_removed_failed".

Checking the logs, it reports that the device does not exist, despite the config and other error messages still reporting it does.

What did you expect to happen

I expected zigbee2mqtt to remove all reference to them and forget them.

How to reproduce it (minimal and precise)

• Physically turn off a CC2530/CC2531 router that was previously paired and in use
• Force remove the device via zigbee2mqtt/bridge/config/force_remove topic

Debug Info

zigbee2mqtt version: 1.12.0 (via hassio Add-on container) CC253X firmware version: CC2531_DEFAULT_20190608 (zStack12, 2.6.3, revision 20190608)

Recent relevant zigbee2mqtt logs

``` zigbee2mqtt:error 2020-04-01 17:21:27: Failed to execute LQI for '0x00124b0014b93ca9' zigbee2mqtt:error 2020-04-01 17:21:37: Failed to execute LQI for '0x00124b0014b93c68' zigbee2mqtt:error 2020-04-01 17:22:02: Cannot force remove, device '0x00124b0014b93c68' does not exist zigbee2mqtt:error 2020-04-01 17:22:28: Failed to execute LQI for '0x00124b0014b93ca9' zigbee2mqtt:error 2020-04-01 17:22:38: Failed to execute LQI for '0x00124b0014b93c68' zigbee2mqtt:error 2020-04-01 17:22:46: Cannot force remove, device '0x00124b0014b93c68' does not exist ```

Config info for the router devices

```json [{ "ieeeAddr":"0x00124b0014b93ca9", "type":"Router", "networkAddress":38273, "model":"CC2530.ROUTER", "vendor":"Custom devices (DiY)", "description":"[CC2530 router](http://ptvo.info/cc2530-based-zigbee-coordinator-and-router-112/)", "friendly_name":"0x00124b0014b93ca9", "manufacturerID":4447, "manufacturerName":"LUMI", "powerSource":"Mains (single phase)", "modelID":"lumi.router", "lastSeen":null }, { "ieeeAddr":"0x00124b0014b93c68", "type":"Router", "networkAddress":12373, "model":"CC2530.ROUTER", "vendor":"Custom devices (DiY)", "description":"[CC2530 router](http://ptvo.info/cc2530-based-zigbee-coordinator-and-router-112/)", "friendly_name":"0x00124b0014b93c68", "manufacturerID":4447, "manufacturerName":"LUMI", "powerSource":"Mains (single phase)", "modelID":"lumi.router", "lastSeen":null }] ```

[Koenkk]

Are the entries still in database.db, could you share this file?

[dortamur]

Yes, they are still in there.

Here's the two entries for the two routers in question (21 & 23), plus a third router still in active use (7), and the controller (1).

{"id":1,"type":"Coordinator","ieeeAddr":"0x00124b0018e1a3d2","nwkAddr":0,"manufId":0,"epList":[1,2,3,4,5,6,11],"status":"online","joinTime":1575970496,"endpoints":{"1":{"profId":260,"epId":1,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"2":{"profId":257,"epId":2,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"3":{"profId":261,"epId":3,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"4":{"profId":263,"epId":4,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"5":{"profId":264,"epId":5,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"6":{"profId":265,"epId":6,"devId":5,"inClusterList":[],"outClusterList":[],"clusters":{}},"11":{"profId":260,"epId":11,"devId":1024,"inClusterList":[],"outClusterList":[1280],"clusters":{"ssIasZone":{"dir":{"value":2},"cmdRsps":{"enrollReq":{"exec":"_exec_"},"statusChangeNotification":{"exec":"_exec_"}}}}}},"_id":"YNct4X3KCdYWFiYz"}
{"id":7,"type":"Router","ieeeAddr":"0x00124b0018e1f850","nwkAddr":57738,"manufId":4447,"manufName":"LUMI","powerSource":"Mains (single phase)","modelId":"lumi.router","epList":[8],"endpoints":{"8":{"profId":260,"epId":8,"devId":256,"inClusterList":[0,6],"outClusterList":[0,6],"clusters":{"genBasic":{"attributes":{"modelId":"lumi.router","manufacturerName":"LUMI","powerSource":1,"zclVersion":1,"hwVersion":1,"dateCode":"2018-09-26"}},"genOnOff":{"attributes":{"onOff":0}}},"binds":[]}},"hwVersion":1,"dateCode":"2018-09-26","zclVersion":1,"interviewCompleted":true,"meta":{},"lastSeen":1580531864152}
{"id":21,"type":"Router","ieeeAddr":"0x00124b0014b93ca9","nwkAddr":38273,"manufId":4447,"manufName":"LUMI","powerSource":"Mains (single phase)","modelId":"lumi.router","epList":[8],"status":"offline","joinTime":null,"endpoints":{"8":{"profId":260,"epId":8,"devId":256,"inClusterList":[0,6],"outClusterList":[0,6],"clusters":{"genBasic":{"dir":{"value":3},"attributes":{"hwVersion":1,"manufacturerName":"LUMI","modelId":"lumi.router","dateCode":"2018-09-26","powerSource":1,"locationDesc":"","physicalEnv":0,"deviceEnabled":1}},"genOnOff":{"dir":{"value":3},"attributes":{"onOff":0}}}}},"_id":"82NZ4w16FsGIV6cr"}
{"id":23,"type":"Router","ieeeAddr":"0x00124b0014b93c68","nwkAddr":12373,"manufId":4447,"manufName":"LUMI","powerSource":"Mains (single phase)","modelId":"lumi.router","epList":[8],"status":"offline","joinTime":null,"endpoints":{"8":{"profId":260,"epId":8,"devId":256,"inClusterList":[0,6],"outClusterList":[0,6],"clusters":{"genBasic":{"dir":{"value":3},"attributes":{"hwVersion":1,"manufacturerName":"LUMI","modelId":"lumi.router","dateCode":"2018-09-26","powerSource":1,"locationDesc":"","physicalEnv":0,"deviceEnabled":1}},"genOnOff":{"dir":{"value":3},"attributes":{"onOff":0}}}}},"_id":"HraCQnRBUv6kFBGK"}

Let me know if you need any more of database.db.

[Koenkk]

Is it also in configuration.yaml?

[dortamur]

They are not mentioned in the devices.yaml, which gets included in the configuration.yaml.

It's been a while since I've tried clearing these short of just the force_remove topic. I may have manually deleted them from devices.yaml some time in the past.

[Koenkk]

They also should be there

[dortamur]

I manually added them back into devices.yaml, restarted, and force_remove now worked as expected. Hurrah!

I was under the impression that devices.yaml would be updated and populated by anything in database.db, but I guess that must only apply to active/found devices.

Might I suggest adding a consistency check comparing devices.yaml and database.db, including offline devices, or making force_remove remove all references from any place, even if it's missing from some expected places. Having a device listed as "there", but failing to force remove is confusing and misleading.

Thanks!

[Koenkk]

What about automatically re-adding devices/groups to configuration.yaml when they exist in database.db?

[dortamur]

That would be fine too. Then there'd be consistency and the force_remove would work.

[Koenkk]

Implemented, thanks!