jneo8
commented
9 months ago (by peter-sabaini) Some notes on this: on an affected system I can see the prometheus-libvirt-exporter regularly timing out on a lock held by a (migration?) process inside libvirtd
Error message:
... Failed to scrape metrics: virError(Code=68, Domain=10, Message='Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3Params)')
There isn't too much the p-l-e can do about this. As peppepetra suggests, we could make the check_http nrpe check configurable with a timeout to reduce false negatives here.
However, independently of the nrep check timeout we will lose libvirt-exporter data precision as during the events when we can't get a libvirtd connection (or it becomes unresponsive) we won't scrape any data either
In overloaded environment
curl http://127.0.0.1:9177/metrics is taking > 30 seconds,
while the check timeouts after 10 seconds (default value) and this parameter is not configurable.
A config option to increase it should be added to avoid false negatives.
Imported from Launchpad using lp2gh.
date created: 2020-06-15T14:36:33Z
owner: peppepetra
assignee: None
the launchpad url