search

pryorda / vmware_exporter

VMWare vCenter Exporter for Prometheus
BSD 3-Clause "New" or "Revised" License
515 stars 199 forks source link

builtins.RuntimeError: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this. #312

Open JefJia opened 2 years ago

JefJia commented 2 years ago

aceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 763, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 858, in _runCallbacks current.result, *args, kwargs File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 1750, in gotResult current_context.run(_inlineCallbacks, r, gen, status) File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 62, in run return f(*args, *kwargs) --- --- File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 1657, in _inlineCallbacks cast(Failure, result).throwExceptionIntoGenerator, gen File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/internet/defer.py", line 62, in run return f(args, kwargs) File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/usr/local/lib/python3.6/site-packages/vmware_exporter-0.18.2-py3.6.egg/vmware_exporter/vmware_exporter.py", line 1942, in _async_render_GET request.finish() File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/web/server.py", line 277, in finish return http.Request.finish(self) File "/usr/local/lib/python3.6/site-packages/Twisted-22.2.0-py3.6.egg/twisted/web/http.py", line 1133, in finish "Request.finish called on a request after its connection was lost; "

trippinnik commented 2 years ago

I see this too. I am presuming it is a retry before first attempt completed as it takes a long time?

laizhimingong commented 2 years ago

Yes, I have encountered this situation many times recently, and the error reports are consistent

crockk commented 2 years ago

Also seeing this, my scrapes usually take under 30s but every now and again it seems like it just randomly times out (scrape interval 1m, timeout 1m), and I see a lot of this error in the logs.

Graph for reference: image

tundeajibade commented 2 years ago

I also see the same error in the logs. Has anyone found a workaround?

trippinnik commented 2 years ago

i moved to this: https://github.com/cblomart/vsphere-graphite instead. More tunable and doesn't reset everytime endpoint is hit for metrics. Seems like this project struggles with big environments.