search

pdreker / fritz_exporter

Prometheus exporter for Fritz!Box home routers
Other
155 stars 32 forks source link

a lot duplicate metrics when FRITZ_HOST_INFO is enabled #115

Closed j0ju closed 1 year ago

j0ju commented 1 year ago

When enabling FRITZ_HOST_INFO the HTTP body gets large but also contains a lot of duplicate metrics. This leads to an increase of factor 10 to 100 in http body size, normal exporter output is approx. 10k to 20k.

# ls -lh exporter-dump-7*
-rw-r--r--    1 root     root      111.8K Dec 29 03:27 exporter-dump-7370
-rw-r--r--    1 root     root      229.4K Dec 29 03:29 exporter-dump-7470
-rw-r--r--    1 root     root        1.0M Dec 29 03:27 exporter-dump-7570

I read the warning in about increased load when enabling FRITZ_HOST_INFO in https://github.com/pdreker/fritz_exporter/blob/develop/docs/configuration.rst The increased load comes from more API calls to the fritzbox for host informations.

This has been observed with exporter version 2.1.6 deployed via docker on the following boxes:

See example from direct curl dump (A bit anonymized: I applied sed -re 's/((serial|hostname|ip_address|mac_address)=)"[^"]+(....)"/\1"...\3"/g' to the output)

# curl $EXPORTER_URL | grep -v "^#" | sort | uniq -c | sort -n
...
     26 fritz_host_speed{hostname="...-k8s",interface="",ip_address="....105",mac_address="...D:0F",model="",port="0",serial="...C3A4"} 0.0
     27 fritz_host_active{hostname="...iene",interface="",ip_address="...3.18",mac_address="...F:A8",model="udhcp 1.30.1",port="0",serial="...C3A4"} 0.0
     27 fritz_host_speed{hostname="...iene",interface="",ip_address="...3.18",mac_address="...F:A8",model="udhcp 1.30.1",port="0",serial="...C3A4"} 0.0
     28 fritz_host_active{hostname="...7490",interface="",ip_address="...23.1",mac_address="...3:A4",model="",port="0",serial="...C3A4"} 1.0
     28 fritz_host_speed{hostname="...7490",interface="",ip_address="...23.1",mac_address="...3:A4",model="",port="0",serial="...C3A4"} 0.0
     29 fritz_host_active{hostname="...mmer",interface="",ip_address="....104",mac_address="...D:39",model="dhcpcd-6.8.2:Linux-3.8.13+:armv7l:Marvell",port="0",serial="...C3A4"} 0.0
     29 fritz_host_speed{hostname="...mmer",interface="",ip_address="....104",mac_address="...D:39",model="dhcpcd-6.8.2:Linux-3.8.13+:armv7l:Marvell",port="0",serial="...C3A4"} 0.0
     30 fritz_host_active{hostname="...eche",interface="",ip_address="....112",mac_address="...C:06",model="",port="0",serial="...C3A4"} 0.0
     30 fritz_host_speed{hostname="...eche",interface="",ip_address="....112",mac_address="...C:06",model="",port="0",serial="...C3A4"} 0.0
     31 fritz_host_active{hostname="...in10",interface="",ip_address="....111",mac_address="...5:93",model="",port="0",serial="...C3A4"} 0.0
     31 fritz_host_speed{hostname="...in10",interface="",ip_address="....111",mac_address="...5:93",model="",port="0",serial="...C3A4"} 0.0
     32 fritz_host_active{hostname="...t-03",interface="",ip_address="...3.83",mac_address="...D:40",model="",port="0",serial="...C3A4"} 0.0
     32 fritz_host_speed{hostname="...t-03",interface="",ip_address="...3.83",mac_address="...D:40",model="",port="0",serial="...C3A4"} 0.0
     33 fritz_host_active{hostname="...t-02",interface="",ip_address="...3.82",mac_address="...5:CB",model="",port="0",serial="...C3A4"} 0.0
     33 fritz_host_speed{hostname="...t-02",interface="",ip_address="...3.82",mac_address="...5:CB",model="",port="0",serial="...C3A4"} 0.0
     34 fritz_host_active{hostname="...t-01",interface="",ip_address="...3.81",mac_address="...F:D6",model="",port="0",serial="...C3A4"} 0.0
     34 fritz_host_speed{hostname="...t-01",interface="",ip_address="...3.81",mac_address="...F:D6",model="",port="0",serial="...C3A4"} 0.0
     35 fritz_host_active{hostname="...dolt",interface="",ip_address="...3.29",mac_address="...2:13",model="",port="0",serial="...C3A4"} 0.0
     35 fritz_host_speed{hostname="...dolt",interface="",ip_address="...3.29",mac_address="...2:13",model="",port="0",serial="...C3A4"} 0.0
     36 fritz_host_active{hostname="...j0ju",interface="",ip_address="....115",mac_address="...A:DC",model="",port="0",serial="...C3A4"} 0.0
     36 fritz_host_speed{hostname="...j0ju",interface="",ip_address="....115",mac_address="...A:DC",model="",port="0",serial="...C3A4"} 0.0
...
pdreker commented 1 year ago

Interesting find, I never actually stumbled across this (or rather: I never noticed the duplicates) o.O

As the exporter does not cache anything, but the output is more or less directly derived from the query results from the box after some remapping this may actually mean, that the exporter also queries the data multiple times, which in turn would of course mean that it is querying the box far too much.

Reproduces with 2.1.6 and current develop.

pdreker commented 1 year ago

OK, I have 83 hosts reported by my FB7590 @7.29 and I get 83 repetitions of 1 entry, 82 repetitions of the next, 81 of the one after that and so on, until the last entry is only repeated once (occurs 2 times).

Smells like a logic error or a misunderstanding how the API works.

pdreker commented 1 year ago

found it... of course it was an indentation error: Lines 865/866 (yield ...) must be unindentend one level, as other wise the metrics array will be returned every time an entry is added, instead of building the array and then returning it in one go.

Unfortunately this does not really improve speed, but will obviously reduce the amount of data transferred by a lot, due to this bug having a quadratic correlation with the number of hosts returned...

I will release an new version shortly.

pdreker commented 1 year ago

Took a bit longer than expected (long overdue CI/CD refactor was in the way), but it is done: 2.2.1 is out and it fixes this bug.

Thanks and have fun!

j0ju commented 1 year ago

Wow that's fast! Thanks for this project and fixing. Tested recently released 2.2.1.