search

fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Other
303 stars 178 forks source link

can not connect to kafka on localhost:9092 #321

Closed ezolnbl-dpc closed 3 years ago

ezolnbl-dpc commented 4 years ago

Problem on windows, everything works fine with kafka. latest version from everything, i can send messages to kafka, i can connect to it but fluent-kafka-plugin can not connect.

<source>
  @type kafka

  brokers 127.0.0.1:9092
  <topic>
    topic something
    partition 0
    offset -1
  </topic>
  get_kafka_client_log true
</source>

<match **>
  @type elasticsearch

  host 127.0.0.1
  index_name deploys
  type_name _doc
  user elastic
  password changeme
  with_transporter_log

</match>
2020-02-18 11:50:43 +0100 [info]: parsing config file is succeeded path="C:\\opt\\td-agent\\etc\\td-agent\\td-agent.conf"
2020-02-18 11:50:44 +0100 [warn]: Consider to specify log_level with @log_level.
2020-02-18 11:50:44 +0100 [info]: using configuration file: <ROOT>
  <source>
    @type kafka
    brokers "127.0.0.1:9092"
    get_kafka_client_log true
    <topic>
      topic something
      partition 0
      offset -1
    </topic>
  </source>
  <match **>
    @type elasticsearch
    host "127.0.0.1"
    index_name "deploys"
    type_name "_doc"
    user "elastic"
    password xxxxxx
    with_transporter_log
  </match>
</ROOT>
2020-02-18 11:50:44 +0100 [info]: starting fluentd-1.7.4 pid=5056 ruby="2.4.9"
2020-02-18 11:50:44 +0100 [info]: spawn command to main:  cmdline=["C:/opt/td-agent/embedded/bin/ruby.exe", "-Eutf-8", "C:/opt/td-agent/embedded/bin/fluentd", "-c", "C:\\opt\\td-agent\\etc\\td-agent\\td-agent.conf", "--under-supervisor"]
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-elasticsearch' version '4.0.3'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-elasticsearch' version '3.5.5'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-kafka' version '0.12.3'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-kafka' version '0.12.1'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.0'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-prometheus' version '1.7.0'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-s3' version '1.2.0'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-td' version '1.0.0'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-webhdfs' version '1.2.4'
2020-02-18 11:50:45 +0100 [info]: gem 'fluent-plugin-windows-eventlog' version '0.4.3'
2020-02-18 11:50:45 +0100 [info]: gem 'fluentd' version '1.7.4'
2020-02-18 11:50:45 +0100 [info]: adding match pattern="**" type="elasticsearch"
2020-02-18 11:50:45 +0100 [warn]: #0 Consider to specify log_level with @log_level.
2020-02-18 11:50:45 +0100 [info]: #0 GET http://elastic:********@127.0.0.1:9200/ [status:200, request:0.014s, query:n/a]
2020-02-18 11:50:45 +0100 [info]: adding source type="kafka"
2020-02-18 11:50:46 +0100 [info]: #0 starting fluentd worker pid=10720 ppid=5056 worker=0
2020-02-18 11:50:46 +0100 [info]: #0 fluentd worker is now running worker=0
2020-02-18 11:50:47 +0100 [info]: #0 New topics added to target list: zeebe-workflow
2020-02-18 11:50:47 +0100 [info]: #0 Fetching cluster metadata from kafka://127.0.0.1:9092
2020-02-18 11:50:47 +0100 [info]: #0 Discovered cluster metadata; nodes: localhost:9092 (node_id=1)
2020-02-18 11:50:57 +0100 [error]: #0 [list_offset] Timed out while trying to connect to localhost:9092: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection fail
ed because connected host has failed to respond.
2020-02-18 11:50:57 +0100 [warn]: #0 Error while fetching messages, Kafka::ConnectionError: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because co
nnected host has failed to respond.; retrying...

localhost is somehow problematic on windows and as i see the connection changed from 127.0.0.1 to localhost at the end: "Timed out while trying to connect to localhost:9092"

ganmacs commented 4 years ago

, i can send messages to kafka,

How did you check it? Can you please give us the specific command or other ways?

repeatedly commented 4 years ago

Please write actual kafka version because kafka sometimes updates their protocol.

ezolnbl-dpc commented 4 years ago

i used the scripts from the kafka distribution: kafka-2.12-2.4.0 kafa-console-producer.sh but i think this has nothing to do with kafka version

repeatedly commented 4 years ago

i think this has nothing to do with kafka version

I confirmed this point. I tested latest fluent-plugin-kafna / kafka 2.4.0 combo and it worked on my Mac.

...snip...
: using configuration file: <ROOT>
  <source>
    @type kafka
    brokers "127.0.0.1:9092"
    get_kafka_client_log true
    <topic>
      topic example
      partition 0
      offset -1
    </topic>
  </source>
  <match example.**>
    @type stdout
  </match>
</ROOT>
2020-02-21 17:50:40 +0900 [info]: starting fluentd-1.9.2 pid=43679 ruby="2.6.3"
2020-02-21 17:50:40 +0900 [info]: spawn command to main:  cmdline=["/Users/repeatedly/.rbenv/versions/2.6.3/bin/ruby", "-Eascii-8bit:ascii-8bit", "/Users/repeatedly/dev/fluentd/fluentd/bin/fluentd", "-c", "in_kafka.conf", "--under-supervisor"]
2020-02-21 17:50:41 +0900 [info]: adding match pattern="example.**" type="stdout"
2020-02-21 17:50:41 +0900 [info]: adding source type="kafka"
2020-02-21 17:50:41 +0900 [info]: #0 starting fluentd worker pid=43693 ppid=43679 worker=0
2020-02-21 17:50:41 +0900 [info]: #0 fluentd worker is now running worker=0
2020-02-21 17:50:42 +0900 [info]: #0 New topics added to target list: example
2020-02-21 17:50:42 +0900 [info]: #0 Fetching cluster metadata from kafka://127.0.0.1:9092
2020-02-21 17:50:42 +0900 [info]: #0 Discovered cluster metadata; nodes: 192.168.11.2:9092 (node_id=0)
2020-02-21 17:52:06.479765000 +0900 example: {"k":"v"}

So if ruby-kafka, in_kafka uses this gem internally, can't connect to your cluster, your environment is something wrong for ruby-kafka.

I didn't hit this problem, so I don't have good advise. Need help from windows and kafka expert.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

github-actions[bot] commented 3 years ago

This issue was automatically closed because of stale in 30 days