joekiller
commented
10 years ago This looks like logstash couldn't talk to elasticsearch. You can see if the Kafka portion is working by making a stdout output and seeing if you see your data. On May 21, 2014 12:32 PM, "Mark Olliver" notifications@github.com wrote:
Hi,
I am using Kafka 0.8.1.1 and Logstash 1.4.1 but it looks like this plugin has issues working with them? I can telnet to both kafka and elasticsearch from the machine that is running this job.
Any ideas?
As far as I can tell I am reading the events from kafka as when i 'ctrl' + 'c' then i do see entries on the console i expected in the stream. Mark
log4j, [2014-05-21T17:24:18.999] WARN: org.elasticsearch.discovery: [l] waited for 30s and no initial state was set by the discovery Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s] at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180) at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492) at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:615) at java.lang.Thread.run(java/lang/Thread.java:744)
logstash.conf
input { kafka { zk_connect => "ZK_SERVER:2181" group_id => "logstash" topic_id => "bid" reset_beginning => false consumer_threads => 1 queue_size => 20 } }
output { elasticsearch { embedded => "false" host => "ES_SERVER" port => 9200 } }
— Reply to this email directly or view it on GitHubhttps://github.com/joekiller/logstash-kafka/issues/21 .
Hi,
I am using Kafka 0.8.1.1 and Logstash 1.4.1 but it looks like this plugin has issues working with them? I can telnet to both kafka and elasticsearch from the machine that is running this job.
Any ideas?
As far as I can tell I am reading the events from kafka as when i 'ctrl' + 'c' then i do see entries on the console i expected in the stream. Mark
log4j, [2014-05-21T17:24:18.999] WARN: org.elasticsearch.discovery: [l] waited for 30s and no initial state was set by the discovery Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s] at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180) at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492) at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:615) at java.lang.Thread.run(java/lang/Thread.java:744)
logstash.conf
input { kafka { zk_connect => "ZK_SERVER:2181" group_id => "logstash" topic_id => "bid" reset_beginning => false consumer_threads => 1 queue_size => 20 } }
output { elasticsearch { embedded => "false" host => "ES_SERVER" port => 9200 } }