search

prometheus-community / node-exporter-textfile-collector-scripts

Scripts for node-exporter's textfile collector
Apache License 2.0
505 stars 188 forks source link

apt info should report cache age #180

Closed anarcat closed 1 year ago

anarcat commented 1 year ago

We cannot alert on stale apt caches. Combined with #179, we can quickly end up in a situation where critical security upgrades are delayed or never installed.

We should have a timestamp metric showing when the last apt update was ran. Optionnally, we could have that metric per mirror as well.

anarcat commented 1 year ago

@julian-klode what would be the canonical way to check when apt update ran last? maybe the last mod time on /var/cache/apt/pkgcache.bin? i guess we could also populate a separate metric based on the /var/lib/apt/lists/*InRelease timestamp, or should we parse the Date: field in those??

thanks, and sorry for the ping if that's inappropriate...

anarcat commented 1 year ago

i picked only pkgcache.bin in #182 for now, with the understanding that it's an important metric to add once we remove the auto-update in #181. This could be extended to cover mirror ages later on, but I suspect that would trigger a lot of noise in a larger fleet, as each server would individually report that its list is out of date... I guess an alerting rule could be tweaked to avoid alerting too often, but still... seems overkill for now.

geor-g commented 1 year ago

apt ships apt.systemd.daily, which gets installed into /usr/lib/apt/. It provides check_stamp() and relies on /var/lib/apt/periodic/update-stamp. I can't tell if it makes sense to rely on this code, but in any case this might provide an idea how to implement this.

anarcat commented 1 year ago

yeah, in #181 i actually document how that works too, or at least an example of it.