Open lairdnote opened 8 years ago
For some reason, hekad can't send data to ES (probably a network issue) so I would recommend to look into the ES logs too or use tcpdump on tcp port 9200. Since you configure the 'block' policy for the buffering, the Heka pipeline gets stuck when the output plugin has filled the local buffer. Hence the idle packs errors.
Next time, please use the mailing list (heka@mozilla.org) for such request.
I checked network . network is fine . the cache data has send to ES . I check ES data. find cache data into . So .I thins maybe ES output plugin can't Recycle cache. I change hekad version to 0.9.2 so all things is good
We use a similar configuration with buffering in our environment and everything works fine for us. What's in your checkpoint.txt file for the Elasticsearch output cache directory?
this is my checkpoint.txt file.
root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# ll total 204812 drwxr--r-- 2 root root 4096 May 20 05:47 ./ drwxr--r-- 3 root root 4096 May 20 04:30 ../ -rw-r--r-- 1 root root 209715062 May 20 05:16 3.log -rw-r--r-- 1 root root 0 May 20 05:23 4.log -rw-r--r-- 1 root root 0 May 20 05:47 5.log -rw-r--r-- 1 root root 11 May 20 05:17 checkpoint.txt root@heka:/var/cache/hekad/output_queue/ElasticSearchOutput# cat checkpoint.txt 3 189562626
thanks for your help
Hi. my log: 2016/05/20 05:26:05 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:26:05 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:26:05 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:26:05 2016/05/20 05:26:35 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:26:35 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:26:35 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:26:35 2016/05/20 05:27:05 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:27:05 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:27:05 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:27:05 2016/05/20 05:27:35 Diagnostics: 1 packs have been idle more than 120 seconds. 2016/05/20 05:27:35 Diagnostics: (input) Plugin names and quantities found on idle packs: 2016/05/20 05:27:35 Diagnostics: ElasticSearchOutput: 1 2016/05/20 05:27:35
my hekad version : heka_0.10.0_amd64.deb
my hekad configre:
[ElasticSearchOutput] message_matcher = "Type =~ /gunicorn.*/" server = "http://192.168.1.12:9200" flush_interval = 200 flush_count = 100 encoder = "ES_Encoder" use_buffering = true http_disable_keepalives = true
[ElasticSearchOutput.buffering] full_action = "block" max_buffer_size = 209715200 #200m cursor_update_count = 1000
The input have 400M size file. when hekad LogstreamerInput was complete text file. so cache dir was full and some record send to ES I wait some time .I see heka dashboard just has ok. but cache dir don't reduce root@lairdnote:/var/log# du -hs /var/cache/hekad/ 205M /var/cache/hekad/
I think this is ESOutput issue .