search

python-diamond / Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
http://diamond.readthedocs.org/
MIT License
1.74k stars 600 forks source link

File Handles Left Open for Logs #720

Open TheConnMan opened 6 years ago

TheConnMan commented 6 years ago

It appears TimedRotatingFileHandler is not handling multiple writer threads well. http://thinlight.org/2011/08/10/python-logging-from-multiple-processes/ is an article highlighting the issue.

I ran a test with the vanilla agent installed on Ubuntu 16.04 on Aug 31st and found that logs are getting rotated, but the original rotated log file is kept open.

root@ip-x-x-x-x:/var/log/diamond# ls -lh
total 25M
-rw-r--r-- 1 root root 1.2M Sep  4 13:04 archive.log
-rw-r--r-- 1 root root 533K Aug 31 23:59 archive.log.2018-08-31
-rw-r--r-- 1 root root 2.1M Sep  1 23:59 archive.log.2018-09-01
-rw-r--r-- 1 root root 2.1M Sep  2 23:59 archive.log.2018-09-02
-rw-r--r-- 1 root root 2.1M Sep  3 23:59 archive.log.2018-09-03
-rw-r--r-- 1 root root 2.5M Sep  4 13:04 diamond.log
-rw-r--r-- 1 root root 1.2M Aug 31 23:59 diamond.log.2018-08-31
-rw-r--r-- 1 root root 4.5M Sep  1 23:59 diamond.log.2018-09-01
-rw-r--r-- 1 root root 4.5M Sep  2 23:59 diamond.log.2018-09-02
-rw-r--r-- 1 root root 4.5M Sep  3 23:59 diamond.log.2018-09-03
root@ip-x-x-x-x:/var/log/diamond# lsof | grep diamond
bash      16131             theconnman  cwd       DIR              202,1     4096     305452 /var/log/diamond
sudo      16189                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
su        16190                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
bash      16191                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
lsof      16635                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
grep      16636                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
lsof      16637                   root  cwd       DIR              202,1     4096     305452 /var/log/diamond
python    30116                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30116                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30127             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30378             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30379             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30406             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30420             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30447             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30117 30473             root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30125                   root    2w      REG              202,1  1173857     305523 /var/log/diamond/archive.log
python    30125                   root    3w      REG              202,1  2571314     305522 /var/log/diamond/diamond.log
python    30128                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30128                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30130                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30130                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30132                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30132                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30135                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30135                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30139                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30139                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31
python    30143                   root    2w      REG              202,1   545051     305510 /var/log/diamond/archive.log.2018-08-31
python    30143                   root    3w      REG              202,1  1194245     305507 /var/log/diamond/diamond.log.2018-08-31