search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.4k stars 531 forks source link

About ElasticSearch OutPut pack can't pack.Recycle . #1938

Open lairdnote opened 8 years ago

lairdnote commented 8 years ago

Hi. my log: 2016/05/20 05:26:05 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:26:05 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:26:05 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:26:05 2016/05/20 05:26:35 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:26:35 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:26:35 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:26:35 2016/05/20 05:27:05 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:27:05 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:27:05 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:27:05 2016/05/20 05:27:35 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:27:35 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:27:35 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:27:35

my hekad version : heka_0.10.0_amd64.deb

my hekad configre:

[ElasticSearchOutput] message_matcher = "Type =~ /gunicorn.*/" server = "http://192.168.1.12:9200" flush_interval = 200 flush_count = 100 encoder = "ES_Encoder" use_buffering = true http_disable_keepalives = true

[ElasticSearchOutput.buffering] full_action = "block" max_buffer_size = 209715200 #200m cursor_update_count = 1000

The input have 400M size file. when hekad LogstreamerInput was complete text file. so cache dir was full and some record send to ES I wait some time .I see heka dashboard just has ok. but cache dir don't reduce root@lairdnote:/var/log# du -hs /var/cache/hekad/ 205M /var/cache/hekad/

I think this is ESOutput issue .

simonpasquier commented 8 years ago

For some reason, hekad can't send data to ES (probably a network issue) so I would recommend to look into the ES logs too or use tcpdump on tcp port 9200. Since you configure the 'block' policy for the buffering, the Heka pipeline gets stuck when the output plugin has filled the local buffer. Hence the idle packs errors.

Next time, please use the mailing list (heka@mozilla.org) for such request.

lairdnote commented 8 years ago

I checked network . network is fine . the cache data has send to ES . I check ES data. find cache data into . So .I thins maybe ES output plugin can't Recycle cache. I change hekad version to 0.9.2 so all things is good

simonpasquier commented 8 years ago

We use a similar configuration with buffering in our environment and everything works fine for us. What's in your checkpoint.txt file for the Elasticsearch output cache directory?

lairdnote commented 8 years ago

this is my checkpoint.txt file.

root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# ll total 204812 drwxr--r-- 2 root root 4096 May 20 05:47 ./ drwxr--r-- 3 root root 4096 May 20 04:30 ../ -rw-r--r-- 1 root root 209715062 May 20 05:16 3.log -rw-r--r-- 1 root root 0 May 20 05:23 4.log -rw-r--r-- 1 root root 0 May 20 05:47 5.log -rw-r--r-- 1 root root 11 May 20 05:17 checkpoint.txt root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# cat checkpoint.txt 3 189562626

thanks for your help