Plenty of physically non-existing "ghost devices"

[bcutter]

Following up on https://github.com/gluap/pyduofern-hacs/issues/34.

• duofern.json contains plenty of non-existing devices (which never existed). The file contains almost 400 lines...

• Using services to clean the config or sync devices don't get rid of them. Every usage of service duofern.ask_for_update with parameter all: true lists all of them, of course there's no answer.

Additonally (the only reason I spotted the "ghost devices" at all) as since 0.5.0 devices are created in Home Assistant (which is a pretty nice innovation by the way, quite helpful for some situations) there are currently two ghost devices created in the device registry:

I have no idea when they started to appear or what might have triggered them. Likely during one of the few updates done in the last weeks they appeared. I could in theory check back a few weeks of existing HA backups and check the duofern.json content and if the first appearance matches the date of a performed duoFERN integration update. There were no changes on the hardware side.

Especially as I'm right before integrating four more shutters I want to clean that mess before, so: I'm looking for a way to get rid of them. How to? Where do they originate from?

My current very limited understanding:

duoFERN stick --> duofern.json --> HA with duoFERN integration --> HA entity registry, HA device registry

[gluap]

As you have manual access the easiest way to get rid of the ghost devices is likely to:

• stop homeassistant
• make a backup copy of duofern.json
• empty the list of devices (removing everything between the square brakets after devices)
• start homeassistant again
• at first all duofern devices will be disabled/unavailable
• calling clean config a few times will force it to write the config to disc and you can see the number of duofern entries increasing in duofern.json (should be just real devices, no ghosts)
• once all devices are there you can restart homeassistant. If some devices seem to "not come back" moving them (manually with the local controls) makes them send messages for homeassistant to receive and detect them.
• devices are available again

so going from this

    "devices": [
        {
            "id": "409f82",
            "name": "409f82"
        },
....
        {
            "id": "7fffec",
            "name": "7fffec"
        }
    ]

to this:

"devices": [ ]

I observe the occasional ghost device, but not hundreds of them. Maybe 3 a year or so. It has been like this for me from the very start - not a sudden influx of ghost devices on my side (yet?).

For me it takes a bit of time with homeassistant running for it to re-add all my devices to duofern.json again. I was able to speed it up by moving the devices. In the end a restart made my (previously disabled) devices active again.

Mind you: creating an empty duofern.json is exactly what clean_config should be doing. literally it empties the config copy in ram and saves it immediately. But it seems that the devices are still in memory make back into the memory-version of the config so fast that they get saved again.

I ran the above process to get rid of my two "ghost" devices.

Warning The above is valid for 0.5.5. With 0.5.6 the clean_config service actually works and you have to call sync_devices to write config instead of the above "clean_config" while re-populating your config.

[gluap]

Regarding where devices come from:

you are mostly right - devices are created by duofernstick when it receives radio messages that appear to be valid by a device id. It occasionally saves the known devices to duofern.json. It also tells homeassitant about new devices and "homeassistant devices" get automatically created for the [physical] devices. I believe that homeassistant may retain its own memory of them so it may be necessary to remove/disable the unwanted devices in homeassistant separately once they are removed from duofern.json. I had that workflow with other integrations that created devices.

[gluap]

I see we already fixed this in april

[bcutter]

Warning The above is valid for 0.5.5. With 0.5.6 the clean_config service actually works and you have to call sync_devices to write config instead of the above "clean_config" while re-populating your config.

Will this be reflected in the docs too?

---

No we did not fix it @gluap. But I ran the clean procedure right now with the result: 1) Cleanup duofern.json done 2) HA restarted, all devices and entities unavailable, duofern.json still clean 3) Tried to trigger populating the duofern.json by moving my covers locally/manually, waited, tried again etc. - nothing happend (duofern.json still clean) 4) Ran service duofern.sync_devices which immediately added all - and ONLY the currently physically really existing - devices again. Devices and entities still unavailable, so step 5 5) Restarted HA once again, now devices and entities are back, duofern.json still without ghost devices - pretty much exactly as you described the whole process.

Summary:

• 94 ghost devices (according to duofern.json) removed successfully 🎉
• Unfortunately I still have two ghost devices in the HA device registry which can not be deleted (deleting devices must be supported by the integration itself):

...and tbh I fear editing the core.device_registry and core.entity_registry files manually, that always was a bad idea. So "90 % fixed" (not completely solved) in terms of "duofern.json is clean, but devices still exist in HA".

And as I will integrate few more devices in the next days I'm curious to see if the ghost devices are coming back.

[gluap]

@bcutter Well for me that's all the fixing that needs to be done on the custom component side - providing tools to maintain the device registry really shouldn't have to be done by individual components. Homeassistant decided that disabling dead devices is enough, so that's the same for me.

[bcutter]

Homeassistant decided that disabling dead devices is enough, so that's the same for me.

Home Assistant a long time ago decided to offer integrations the possibility to delete devices to even walk that last mile. No need to keep plenty of disabled but dead devices AND entities anymore. Two quick examples:

a) MQTT

b) iBeacon Tracker

That whole legacy duofern.json file handling not in sync with the HA device registry obviously IS an issue. You're the only one able to fix this so it's up to you - just wanted to point out there are ways (and a need).

[gluap]

@bcutter Thanks for the pointer, seems you just have to tell homeassistant as a component that you're ready to delete. I added this. Apologies, I misunderstood you to be asking for me to write code to programmatically change the homeassistant device database files which I'm not really comfortable doing.

I'd like to add that this is github so anyone is able to make a pr. And I promise: If it works I'll merge it.

[bcutter]

No need to re-invent things provided natively by HA 😃

Of course it is GH, unfortunately there don't seem to be many users with dev skills around duofern in general. So I'm always counting on you 😆

With https://github.com/gluap/pyduofern-hacs/releases/tag/v0.5.11 it seems you already implemented this? Wow, awesome 👍

I'm curious to test this asap.

Off-topic (but important) on

remove non-working sync device command and document.

that service did not work? Can't actually remember. BUT according to my own device setup and integration chcklist after using duofern.start_pairing it was actually needed to use that duofern.sync_devices service for devices / entities to actually get created in HA. Did that change? As mentioned I'm about to integrate few more duoFern devices shortly... so therefore just wondering.

Edit: found https://github.com/gluap/pyduofern-hacs/commit/d19da5465ce9969f8926557d030046b19859d1e3 and really wondering if this is the way to go... I need to restart HA to grab new devices now? Oh well that feels like a step backward TBH. I really prefer a service to take care of that (if not done automatically what is the default behaviour of modern integrations - but I understand pyduofern seems to have a quite rusty base so probably those are workarounds to fit that base into the modern HA, aren't they?). 😲

[gluap]

@bcutter Yeah, I also prefer homeassistant to take care of that but it's not worked for a long time due to changes in how devices are registered in homeassistant. Installing new shutters is something one does only do very rarely, and restarting homeassistant only adds like a minute at most to that procedure, so instead of a non working service with the empty promise that it fixes things I think its best for the documentation to just be honest about it.

[bcutter]

due to changes in how devices are registered in homeassistant.

Maybe we should adopt to this change? Otherwise this for sure is a showstopper for a future v1.0 or even becoming an official integration.

I think its best for the documentation to just be honest about it.

I find that service still quite a lot in the docs (readme).

[gluap]

I added sync_devices back with reduced functionality. no real need for it anymore with the reload integration facility though. Basically what it was doing was what reload integration is doing now.

I do not really see a v1.0 in the future - there's just too many loose ends

Same for official integration. I can't really provide the service level I'd expect from an official HA integration.