search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.39k stars 531 forks source link

LogstreamerInput: 100% CPU idle usage with compressed files #1899

Open smalenfant opened 8 years ago

smalenfant commented 8 years ago

I'm running into this issue with a single gzip file in the file_match in a configuration. CPU goes to 100% when the file has been completely read. 

[custom_ats_2]
type="LogstreamerInput"
decoder = "ATS_transform_decoder"
log_directory = "/tmp/logs"
file_match = "custom_ats_2.log.20160301.02h14m21s-20160301.07h00m32s.old.gz"

I renamed the file, CPU problem gone instantly.

Then I uncompressed the file and added to file_match, erased the registry and restarted. Once the file was completed, the load went down to nothing.

This should confirms there is a problem with compressed (.gz) files and rescanning for changes.