search

python-diamond / Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
http://diamond.readthedocs.org/
MIT License
1.74k stars 599 forks source link

SyncManager memory leak #316

Open loood opened 8 years ago

loood commented 8 years ago

We're running diamond with custom collectors for haproxy and are starting to see the syncmanager process leak memory. It'll continue to leak memory until 100% of memory. Can collectors cause syncmanager to leak memory or ?

ubuntu trusty 14.04.3:$ ps -uwwwp 35672,35673,35676,35684,35687,35689,35694,35698,35701,35706,35714,35722,35727,35732,35736 --forest
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     35672  0.0  0.0   4448   780 ?        Ss   07:22   0:00 /bin/sh -e -c python /usr/local/bin/diamond --foreground /bin/sh
root     35673  0.0  0.0 138384 25564 ?        S    07:22   0:01  \_ diamond                                   
root     35676  2.0  4.7 3920652 3130856 ?     Sl   07:22   0:43      \_ diamond - SyncManager                     
root     35684  1.8  0.0 158336 41036 ?        S    07:22   0:38      \_ diamond - Handlers                        
root     35687  0.0  0.0 137616 20548 ?        S    07:22   0:00      \_ diamond - VMStatCollector                 
root     35689  0.0  0.0 137872 21048 ?        S    07:22   0:02      \_ diamond - NetworkCollector                
root     35694  0.0  0.0 137872 21088 ?        S    07:22   0:00      \_ diamond - CPUCollector                    
root     35698  0.0  0.0 137872 20736 ?        S    07:22   0:00      \_ diamond - MemoryCollector                 
root     35701  0.0  0.0 138128 20760 ?        S    07:22   0:00      \_ diamond - LoadAverageCollector            
root     35706  0.0  0.0 138128 21224 ?        S    07:23   0:00      \_ diamond - IPCollector                     
root     35714  0.0  0.0 138128 20968 ?        S    07:23   0:00      \_ diamond - DiskSpaceCollector              
root     35722  0.1  0.0 138384 21248 ?        S    07:23   0:03      \_ diamond - DiskUsageCollector              
root     35727  4.4  0.0 146220 29392 ?        S    07:23   1:36      \_ diamond - HAProxySocketCollector          
root     35732  4.5  0.0 146792 29828 ?        S    07:23   1:37      \_ diamond - HAProxy2Collector               
root     35736  0.1  0.0 138384 21668 ?        S    07:23   0:02      \_ diamond - TCPCollector                    
ubuntu trusty 14.04.3:$ python -V
Python 2.7.6
ubuntu trusty 14.04.3:$ pip freeze
apt-xapian-index==0.45
chardet==2.0.1
configobj==5.0.6
diamond==4.0.195
docopt==0.6.2
iotop==0.6
pbr==1.8.1
Pillow==2.3.0
psutil==3.2.2
pycrypto==2.6.1
pycurl==7.19.3
pynetlinux==1.1
python-apt===0.9.3.5ubuntu1
python-debian===0.1.21-nmu2ubuntu2
python-ldap==2.4.10
python-memcached==1.53
PyYAML==3.10
requests==0.14.2
send-nsca==0.1.4.1
setproctitle==1.1.9
simplejson==3.3.1
six==1.10.0
ssh-import-id==3.21
ssh-ldap-pubkey==0.4.0
stevedore==1.9.0
urllib3==1.7.1
virtualenv==13.1.2
virtualenv-clone==0.2.6
virtualenvwrapper==3.7.1
ubuntu trusty 14.04.3:$ uname -a
Linux lb-edge-8c6ce0 4.1.8-040108-generic #201509211431 SMP Mon Sep 21 18:34:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
jraby commented 8 years ago

What output handlers are you using and are you seeing data in all of them? I was tracking a memory usage issue tonight and found the culprit to be an output handler that couldn't dump its data fast enough.

loood commented 8 years ago

Only have graphitehandler. I ended up rewriting the collector which seem to stop the memory leak.

josegonzalez commented 7 years ago

@loood do you have a pull request for the GraphiteHandler?