inovex / prometheus-libvirt-exporter

prometheus-libvirt-exporter
MIT License
22 stars 6 forks source link

Failed to collect some domain info while migrating vm #53

Open laushunyu opened 4 hours ago

laushunyu commented 4 hours ago

I use openstack 23.2, and deployed prometheus-libvirt-exporter by apt install prometheus-libvirt-exporter (version 1.6.0).

I found some journal err log while live-migrate vm:

Nov 15 07:49:48 xfusion-3 prometheus-libvirt-exporter[4037]: ts=2024-11-15T07:49:48.773Z caller=prometheus-libvirt-exporter.go:461 level=warn warn="failed to get DomainBlockStats" domain=instance-00001045 msg="Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)"
Nov 15 07:49:48 xfusion-3 prometheus-libvirt-exporter[4037]: ts=2024-11-15T07:49:48.773Z caller=prometheus-libvirt-exporter.go:445 level=warn warn="failed to collect some domain info" domain=instance-00001045 msg="Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)"
Nov 15 07:49:48 xfusion-3 prometheus-libvirt-exporter[4037]: ts=2024-11-15T07:49:48.805Z caller=prometheus-libvirt-exporter.go:575 level=warn warn="failed to get DomainMemoryStats" domain=instance-00001045 msg="Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)"
Nov 15 07:49:48 xfusion-3 prometheus-libvirt-exporter[4037]: ts=2024-11-15T07:49:48.805Z caller=prometheus-libvirt-exporter.go:445 level=warn warn="failed to collect some domain info" domain=instance-00001045 msg="Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params)"

Prometheus show there are some failed metric points:

image

frittentheke commented 3 hours ago

This is somewhat of a known issue which I started a discussion about on the ML a while back: https://lists.libvirt.org/archives/list/users@lists.libvirt.org/thread/PXKLIU7DPF6MDGEHWQYX3KLGCDNRAIW3/