New Fritzbox tools

[Smandurlo]

The problem

Hello, I have a problem with the new 2021.5.0 version and device tracking with fritzbox. Now it relies with a gui integration AVM FRITZ!Box Tools (which is overwhelming for my needs and I don't get why I am forced to use it). I have 4 fritzbox in a mesh configuration and the auto discovery continues to find a new device even if I configured the master (you don't really need to configure the slaves and you cannot ignore it). Moreover, the device tracking service ignores my yaml configuration and duplicates every single device connected, including the one I don't need to monitor. I can disable the one I don't need (extra work), but I cannot configure the extra option like to ignore the new devices (do I really need to disable every new device in the gui?) and consider_home. It is quite a tragedy for me...

HACS shows an error in the legacy fritz module.

I am fine with using a GUI configuration, but it must use the same setting as yaml. Now I cannot use it anymore, every new device in the guest network (I have a lot of them) create a new device tracking that I have disable manually and I get false device tracking alerts because it is triggered immediately ignoring the consider_home setting.

What is version of Home Assistant Core has the issue?

2021.5.0

What was the last working version of Home Assistant Core?

2021.4.6

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

FRITZ!Box Tools

Link to integration documentation on our website

https://www.home-assistant.io/integrations/fritz/

Example YAML snippet

ignored by the gui setting

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant[bot]]

fritz documentation fritz source _{^{(message by IssueLinks)}}

[frenck]

Moreover, the device tracking service ignores my yaml configuration and duplicates every single device connected, including the one I don't need to monitor. I can disable the one I don't need (extra work), but I cannot configure the extra option like to ignore the new devices (do I really need to disable every new device in the gui?) and consider_home.

YAML configuration for integrations connecting to devices or services is going away. This is a strategy the Home Assistant is following for quite some time now and is architecturally decided upon.

As for the option to ignore new entities created, you can enable that option on every integration:

HACS shows an error in the legacy fritz module.

That is a custom integration and not something the Home Assistant project supports or can help you with.

[Smandurlo]

Hi Frenck, thanks for you reply and your great work. I agree with you that the yaml is going away and I am fine with it, but the gui should integrate the same options as the yaml (and the documentation for it).

Thanks for the ingnoring new device tip, but there is still the problem of the consider_home and the autodiscovery for slave fritzboxes inside the network.

The HACS error appears when using the 2021.5.0 core, I don't even have any fritz legacy integration in hacs

[frenck]

but the gui should integrate the same options as the yaml

That is not a rule we have and some options we used to have, we actually don't accept anymore.

The other issues I leave open, as I don't have experience with Fritz! itself.

[cyberjunky]

"It is quite a tragedy for me..."

Same here, very confusing...

[frenck]

Same here, very confusing...

Let's not just do "Same here" or "Me too" answers, instead give the issue a 👍 or provide constructive comments with help and approaches. Thanks 👍

[cyberjunky]

Stating that I have the same issue is somewhat constructive I think? Why is is called Tools why it actually is a device tracker/presence detector if I understand correctly looking at the documentation. One think I miss is a link on the discovered tile to jump to documentation page.

[frenck]

I'm not sure how that is related to this issue report @cyberjunky ?

[Smandurlo]

my2cents: - autodiscovery should be an option to set it on/off or at least add the option to ignore something discovered that you don't want to configure (like my 3 slave fritzboxes) - keep the simple device tracker AND the fritzbox tools, the fritzboox tools is way too much if you just need to track the devices and it doesn't work as expected (like any other device track integration). Actually the device tracking is not working as expected and the discovery service keeps continue to discovery something you don't have to configure (they don't give any information, everything is done by the master).

[frenck]

There is an ignore button to hide the ones you don't want 🤷 (and again, not related to the issue presented in the OP).

[Smandurlo]

Frenck, maybe I was not clear: I understood I can hide the device tracking I don't want to track, but there is not any button to hide the new integration the autodiscovery find! It find a new device, I add it. Done. My fritzbox 192.168.178.1 is ok (with the problem in tracking devices without any consider_home option as any other device tracking service). Then I reboot HA and the auto discovery find another new device, another new fritzbox (a slave in my mesh network). No need to add the integration for it, but there is not the ignore button.

Sorry for misspelling your name in my first reply

[chemelli74]

Then I reboot HA and the auto discovery find another new device, another new fritzbox (a slave in my mesh network). No need to add the integration for it, but there is not the ignore button.

Can you show me the integration page with the discovered device ? there should be a ignore button there. If not we need to understand why.

Simone

[chemelli74]

keep the simple device tracker AND the fritzbox tools, the fritzboox tools is way too much if you just need to track the devices and it doesn't work as expected (like any other device track integration).

The integration will soon have more entities ( sensors, switches, services, ...). If you don't like them, just disable them ;-)

[Idcsl]

The new tools integration seems to work well for me but with two issues.

First is my logs are filling up as every device on the system is now being tracked since there appears no way to disable device tracking for individual devices.

Second issue is the inability to set a picture file to a tracked device as you could in the previous yaml configuration.

[Smandurlo]

chemelli74, it just doesn't work as any other device tracker and it shoudn't be used for it unless it has the same options as any other device tracker. I had to revert to 2021.4.6 and I will stick to it since with your tools the device trackers are not working as I need. What is the point to have billion of options and disable all of them if I need only the device tracker? Maybe you don't understand the problem: I have a lot of devices connecting to my guest wifi, they generates a new device tracker that I don't need to track. I can set up in the gui not to track the new devices, but if I need to add a new device with tracking? Plus, the device tracker has the "consider_home" option to filter the wrong tracking (in a big mesh network it happens very often, I was spammed of false alarms in a couple of hours). I really don't understand why we can't keep the normal device tracking as before AND your fritzbox tools for people that need all that fancy options. If something is not clear, you can always write to me in private in Italian. Thanks.

[drbiosignals]

Unfortunately I can also confirm that the new integration breaks the very reliable device_tracking we had until this update. For people with huge networks the integration creates plenty of device_tracking entities, which should be manually disabled. Moreover, without the consider_home option the device tracking doesn't work as expected. I personally changed to the PING option for device_tracking.

[Smandurlo]

The PING is also not reliable, some mobile phones stop to respond to the ping (iPhone and xiaomi in my experience), the same for the nest thermostat plus I don't see the reason to send a ping to all my network for no reason.

[chemelli74]

First is my logs are filling up as every device on the system is now being tracked since there appears no way to disable device tracking for individual devices.

Please open a separe issue for this problem.

Second issue is the inability to set a picture file to a tracked device as you could in the previous yaml configuration.

This is a feature request, please open a thread here

[chemelli74]

If something is not clear, you can always write to me in private in Italian.

Got your point, but I think it's a bit out of sync. If you want to discuss, you find me on discord.

Simone

[Idcsl]

First is my logs are filling up as every device on the system is now being tracked since there appears no way to disable device tracking for individual devices.

Please open a separe issue for this problem.

Second issue is the inability to set a picture file to a tracked device as you could in the previous yaml configuration.

This is a feature request, please open a thread here

Ok thanks, I have now opened a feature request for the picture file issue and will open a separate issue for the tracked device disable requirement.

[Smandurlo]

I was forced to upgrade to 2021.5.1 because of the broken Netatmo integration. I can confirm the device tracking is broken and useless. I will make a list of problems: 1) not possible to disable the discovered new integration (multiple fritzbox in the mesh network): https://i.postimg.cc/P5WsT7Tm/2021-05-08-2.png 2) an error into the HACS integration: https://i.postimg.cc/xTHTNt9d/2021-05-08-1.png (and the previous image). I don't use any Fritz custom integration with HACS; 3) missing options for device tracking: - ignore new devices (can be done with the gui, but then you cannot add any other new device that you want to track); - consider_home to prevent false alarms.

I still think we shoudn't be forced to use this tool if we just need to track only few devices. The previous integration was really reliable. PING and NMAP are not good with a lot of devices/networks.

Can we, at least, have a custom integration to load with HACS of the previous fritz device tracker?

I have tons of guests devices and time to time new devices I want to track. I cannot do it now.

[da-anda]

I am in the same boat. I have lots of devices in my network (which spans over 3 house holds, having dozens of "smart" devices, PCs etc on the network) and the sheer flood of imported device trackers is super annoying. In the known_devices.yaml there was a flag if a device should be tracked - it would have been nice if this flag would have been respected when the old yaml config got migrated to config flow. Also, not all devices in this known_devices.yaml got imported. The FritzBox is only remembering devices for so long, and now with COVID, friends haven't been over for a long time and the FritzBox doesn't know their smartphones anymore, but they are still in the known_devices.yaml and thus tracking with the old integration would still have worked. So please also import those during migration. Thanks.

I can also second the problem of how to manage which devices should be tracked. If you disable the option to automatically add new devices, you literally have no way of manually adding a new device. So my proposal for an improved config flow would be:

Onboarding

During onboarding, ask if new devices should be automatically added

• if YES: do what is done now and add every device
• if NO: list all currently known devices to the FritzBox with an option to check/select the devices to import

Maintenance

Once the integration is set up and users have the "auto add new devices" setting turned off, there should be an option "add new device" which again provides a list of known devices to the FritzBox with an option to check/select the devices to import

As for the general structure of the integration. I am using the HACS version of FritzBoxTools for over a year now (wifi toggle switches etc) and it's nice. My understanding is that this core integration will integrate these features on the long run, which IMO is nice. I don't see an issue having those features merged with the device tracking, as long as you can still enable/disable the feature set of the integration you like to use, like it is the case with the HACS version. What I do not necessarily like and think is a bad design choice, is that each tracked device is added as a HA device with a device tarcker entity. It IMO would be better if only one Fritz!Box Device Tracker device got added to HA with several device tracker entities.

edit: btw - it might make sense to only use WIFI connected devices for device tracking as I am pretty sure that wired connections do not tend to get moved and need tracking. So if the FritzBox is providing the info if the device is wired/wireless connected, it might be an idea to make use of it. Thank.

[Smandurlo]

the import doesn't care of anything. Actually there is no import, it just make a new list of devices from scratch with the device list in the fritz's network list. My guests network is shared with 2 short rental flats and the list of them gets really huge in a short while. EDIT: It is important to keep the ethernet tracking to know if they get broke/stolen (e.g my TVs in the rental flats are wired, some of my slave mesh fritz are wired and so on).

[chemelli74]

I will make a list of problems:

1. not possible to disable the discovered new integration (multiple fritzbox in the mesh network): https://i.postimg.cc/P5WsT7Tm/2021-05-08-2.png
2. an error into the HACS integration: https://i.postimg.cc/xTHTNt9d/2021-05-08-1.png (and the previous image). I don't use any Fritz custom integration with HACS;

Both are related to the fact that you have a config based on YAML. After first run, config is imported. Then you should remove those lines from configuration.yaml.

Please test and report back. If problem persist, open a separate issue for that.

3. missing options for device tracking: - ignore new devices (can be done with the gui, but then you cannot add any other new device that you want to track); - consider_home to prevent false alarms.

You can disable the new devices or each discovered entities.

AFAIK consider_home is not implemented in the new ScannerEntity

Simone

[drbiosignals]

Unfortunately after testing for some days I can confirm that the device tracking is very unreliable and sometimes doesn’t actualise to the not_home state at all. I am now using the following workaround: For non-phones I am using ping as binary_sensor in order to test online status. For phones I am using iobroker and the tr-064 adapter, which sends the state to the mqtt server and I am finally importing this state to hass using the mqtt presence_detection integration.

[Smandurlo]

Please test and report back. If problem persist, open a separate issue for that.

fixed

1. missing options for device tracking: - ignore new devices (can be done with the gui, but then you cannot add any other new device that you want to track); - consider_home to prevent false alarms.

You can disable the new devices or each discovered entities.

I cannot disable each discovered entity, I have 20-30 new devices connecting to the guests network every week! And if I disable the auto add entity, I cannot add anything else I want to track in the future. We are blaming the same thing and you always reply the same way. It is not really helpful.

AFAIK consider_home is not implemented in the new ScannerEntity

You can always add it, honestly I don't understand why to remove such an useful feature. The device tracker integration has it: https://www.home-assistant.io/integrations/device_tracker/

[Smandurlo]

For non-phones I am using ping as binary_sensor in order to test online status.

the problem is, sometimes, they don't reply fast enough to the ping or they don't reply at all (nest thermostat for exaple) or they occasionally get disconnect and reconnect by themself (or while you simply reboot them). In that moment they are not_home for the ping, but home for the router. The tracking with the router was perfect with the consider_home option.

For phones I am using iobroker and the tr-064 adapter, which sends the state to the mqtt server and I am finally importing this state to hass using the mqtt presence_detection integration.

That is a very a creative way... I hope we will get back the easy, flexible and reliable way as before.

[chemelli74]

Unfortunately after testing for some days I can confirm that the device tracking is very unreliable and sometimes doesn’t actualise to the not_home state at all.

This is very interesting feedback as the integration rely on fritzconnection lib as before.

Simone

[Smandurlo]

I am working on a fritzbox presence detection based on a node for nodered since I don't think we will have a working one as before.

[chemelli74]

I am working on a fritzbox presence detection based on a node for nodered since I don't think we will have a working one as before.

@Smandurlo, help me investigating and fix the new core integration ;-) I'm available on Discord : chemelli74#2180

I will add consider_home later this week hopefully. About reliability, I cannot find any difference honestly; library is the same, calling the same FritzHost function.

Simone

[chemelli74]

I am working on a fritzbox presence detection based on a node for nodered since I don't think we will have a working one as before.

@Smandurlo, help me investigating and fix the new core integration ;-) I'm available on Discord : chemelli74#2180

Please check https://github.com/home-assistant/core/issues/50388#issuecomment-837014426

Simone

[Borcon]

After the update to 2021.5.0 i had the same issue that the device tracking did not work and i had two fritzbox tools visible in the integration. I could solve that issue with the following steps:

• Remove the fritz yaml code
• Remove the fritzbox tools from integration
• Restart HA
• Add fritzbox tools again

The ability in the old integration to track only individual devices was great. Maybe you can make that possible again with the new core integration. In the meantime, I have deactivated new added devices with system options.

@chemelli74 Thanks for the integration and your support.

[mfaber]

About reliability, I cannot find any difference honestly; library is the same, calling the same FritzHost function.

Simone

I found that devices that disconnect from guest wifi do not go back to not_home or unavailable status. They remain in home status indefinetely although fritzbox recognized the signoff (can see in frizbox UI). Restarting the integration has no effect on the device status. After restarting home assistant with the device signed off, the status is correctly regognized as unavailable. With the standard (not guest) wifi, home and not_home detection works fine here (Fritz 7490 with 7.26 firmware). Not a big programmer myself but willing to test and report back if that helps. Thanks!

[Smandurlo]

The tracking of the non guest devices (wifi and ethernet) is the same as before, the problems are the lack of consider_home (which brings to false not_home/home triggers) and the impossibility to select the devices you want to track after you select the "don't track new devices" in the gui. I didn't check the guests wifi tracking.

@chemelli74 thanks for you support, but I don't use discord. Maybe we can use this post so other people will report their feedback

[chemelli74]

PR ready: Add consider_home option to Fritz device_tracker

Simone

[Smandurlo]

good news! I had to turn off all my notification since the upgrade because of this missing feature. Waiting for the better track of new devices. Thank you so much for your work