search

python-diamond / Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite (and others). It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.
http://diamond.readthedocs.org/
MIT License
1.74k stars 601 forks source link

[enhancement] ArchiveHandler: log compression #692

Open bengaywins opened 6 years ago

bengaywins commented 6 years ago

I don't really understand why it wasn't built in, but this handler should compress the old logs. 

6.9G Dec 29 11:47  archive.log.2017-12-27
394M Dec 29 11:47  archive.log.2017-12-27.gz

Having a log this size uncompressed, and more than one, is just pointless and dumb. This is the perfect candidate for compression from this large number to a significantly smaller number. As one sees with the same file compressed with the highest compression (-9) above in gzip.

DStape commented 6 years ago

Hi @gehzumteufel,

Looking at the ArchiveHandler code, it looks like it does some kind of rotation using Python's logging.handlers.TimedRotatingFileHandler. When instantiating, the handler code passes an encoding. By default, this is empty but it can be configured as part of the handler's config.

Could you try setting encoding = bz2 in the config and see if that works? Thanks.

DStape commented 6 years ago

(Just to note, encoding is listed as a config option in the documentation (https://diamond.readthedocs.io/en/latest/handlers/ArchiveHandler/), but maybe we should update that to include examples to make what it does more obvious.

bengaywins commented 6 years ago

@DStape I am not in an environment where I have to worry about this right now (or am even using it for the moment), but @jasonhancock may be able to test it as he is in the same environment I was in still. He's definitely got some logs he could use this on.

jasonsattler commented 6 years ago

Looks like the encoding option worked. I think just adding a quick description and valid values would help.