search

canonical / prometheus-openstack-exporter

OpenStack exporter for the prometheus monitoring system
GNU General Public License v3.0
128 stars 113 forks source link

Server in deadlock #112

Closed drencrom closed 1 year ago

drencrom commented 2 years ago

Hi, I found a deadlock condition in a prometheus-openstack-exporter server running the last snap version. The error is the same as described here for swift with just some line number differences as the server is using python 3.8. This is the stack trace for the error I saw. Could it be the same issue?

johnlettman commented 2 years ago

I have encountered this bug today as well. It appears as a full deadlock of the prometheus-openstack-exporter service with the following traceback in the system journal:

Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]: Traceback (most recent call last):
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:     timer()
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:     cb(*args, **kw)
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]:     waiter.switch()
Jul 13 15:58:21 juju-733f2f-4-lxd-13 prometheus-openstack-exporter.prometheus-openstack-exporter[1683719]: greenlet.error: cannot switch to a different thread

This is prometheus-openstack-exporter snap version 0.1.7 revision 34 inside charm revision 16.

Presently, the workaround is restarting the prometheus-openstack-exporter service (eg. sudo systemctl restart snap.prometheus-openstack-exporter.prometheus-openstack-exporter.service in the case of the charm).

Searching back, BootStack encountered this on a separate environment on Sunday:

REDACTED-prometheus-openstack-exporter-0-prometheus_openstack_exporter_http, this issue happens again, restarting prometheus-openstack-exporter service

2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]: Traceback (most recent  call last):
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:     timer()
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:     cb(*args, **kw)
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:   File "/snap/prometheus-openstack-exporter/34/lib/python3.8/site-packages/eventlet/semaphore.py", line 147, in _do_acquire
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]:     waiter.switch()
2022-07-11T03:01:55Z prometheus-openstack-exporter.prometheus-openstack-exporter[12563]: greenlet.error: cannot switch to a different thread
drencrom commented 2 years ago

Fixed in #115