search

pryorda / vmware_exporter

VMWare vCenter Exporter for Prometheus
BSD 3-Clause "New" or "Revised" License
514 stars 198 forks source link

memory leak when vcenter not reachable #282

Open finkr opened 3 years ago

finkr commented 3 years ago

Thanks to the exporter monitoring feature introduced in https://github.com/pryorda/vmware_exporter/pull/128, I noticed a memory leakage when the exporter can't reach the vCenter. The leaked memory was recovered when the connection resumed (garbage collection).

This occurred when experimenting vmware_exporter with an updated version of the library Prometheus client version 0.11.0.

image

The leakage has to be confirmed with the current version setup (vmware_exporter with client_python 0.0.19).

RyanW8 commented 3 years ago

Screenshot 2021-08-09 at 13 42 10 Can confirm that we see the exact same behaviour, although the vCenter is reachable scraping metrics takes some time. And if it exceeds the scrape timeout this causes the memory to increase

pryorda commented 3 years ago

@RyanW8 What version?

daverstam commented 2 years ago

I had a lot of issues where the exporter would run into race conditions where scrape_timeout would cause the exporter to timeout however the connections seemed to stay and causing the memory to raise.

However changing the threads for reactor/twisted made it much more stable for our environment. The exporter rarely ever exceeds the scrape_timeout (118s) anymore. I'm running the exporter in k8s if that makes any difference. I changed suggestedThreadPoolSize from 25 to 1 and that actually lowered the scrape duration time.

reactor.suggestThreadPoolSize(1)

Our environment with approx 200 hosts and ~ 2000vm's takes around 30s to scrape (including datastores)

thuvh commented 2 years ago

@pryorda should we return empty page when vcenter is not reached