search

jbvmio / kafkactl

CLI for Apache Kafka Management -
http://www.jbvm.io/kafkactl
Apache License 2.0
88 stars 9 forks source link

client/metadata got error from broker -1 while fetching metadata #9

Open Hubbitus opened 4 years ago

Hubbitus commented 4 years ago

Hello.

I am newbie and try to use kafkactl:

podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/etc/kafkactl/config.yml \
    docker.io/deviceinsight/kafkactl:v1.11.1 \
        kafkactl --verbose get topics
[kafkactl] 2020/09/22 09:02:18 Using config file: /etc/kafkactl/config.yml
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[kafkactl] 2020/09/22 09:02:18 Assuming kafkaVersion: 2.0.0
[sarama  ] 2020/09/22 09:02:18 Initializing new client
[sarama  ] 2020/09/22 09:02:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:02:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44214->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:02:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:02:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:02:48 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama  ] 2020/09/22 09:02:48 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:02:48 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:18 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44222->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:18 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:18 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:18 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama  ] 2020/09/22 09:03:18 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:18 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:03:48 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44228->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:03:48 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:48 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:03:48 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:03:48 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama  ] 2020/09/22 09:03:49 client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:03:49 Connected to broker at ecsc00a060af.epam.com:9092 (unregistered)
[sarama  ] 2020/09/22 09:04:19 client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:44244->10.6.194.60:9092: i/o timeout
[sarama  ] 2020/09/22 09:04:19 Closed connection to broker ecsc00a060af.epam.com:9092
[sarama  ] 2020/09/22 09:04:19 client/metadata no available broker to send metadata request to
[sarama  ] 2020/09/22 09:04:19 client/brokers resurrecting 1 dead seed brokers
[sarama  ] 2020/09/22 09:04:19 Closing Client
failed to create cluster admin: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Config is very simple (sed to show just non-comments and non-empty lines):

$ sed 's/#.*$//;/^$/d' .kafkactl.yaml
contexts:
  sandbox:
    name: sandbox
    brokers:
      - ecsc00a060af.epam.com:9092

current-context: sandbox

And according to wireshark dump TCP session kafkactl send metadata request and can connect to the server, but newer receive response: image

Where I am wrong?

jbvmio commented 4 years ago

@Hubbitus - you're using the wrong kafkactl. In your output, deviceinsight/kafkactl, if you want to use this implementation, please try jbvmio/kafkactl.

Hubbitus commented 4 years ago

I'm very sorry for the mistake. And thank you for the response.

Meantime it changes almost nothing. Your image requires config in single place versus search in some common, but then failed with the same error without details:

$ podman run -it --rm --name kafkactl -v ~/.config/kafkactl/config.yml:/root/.kafkactl.yaml \
        docker.io/jbvmio/kafkactl:v1.0.25 \
                --verbose get topics
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:21Z] Initializing new client                      
INFO[2020-09-23T09:03:21Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:22Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:27Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54082->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:27Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:27Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:27Z] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2020-09-23T09:03:27Z] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:27Z] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2020-09-23T09:03:32Z] client/metadata got error from broker -1 while fetching metadata: read tcp 10.0.2.100:54088->10.6.194.60:9092: i/o timeout 
INFO[2020-09-23T09:03:32Z] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2020-09-23T09:03:32Z] client/metadata no available broker to send metadata request to 
INFO[2020-09-23T09:03:32Z] client/brokers resurrecting 1 dead seed brokers 
INFO[2020-09-23T09:03:32Z] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
Hubbitus commented 4 years ago

And it looks like kafkactl request and receive brokers metadata, but stops on that: image Please compare with successful session list topic from kafkacat: image

jbvmio commented 4 years ago

What version of Kafka are your brokers running @Hubbitus ? Can you try adding the --version 1.0.0 flag? Curious have you tried running the direct binary instead of Docker?

Hubbitus commented 4 years ago

I'm unsure about the remote broker's version (waiting answer). Use binary of kafkactl compiled locally changes nothin as I see. --version 1.0.0 also does not helps.

Still, first metadata comes and then tons of TCP retransmissions only: image

zotona commented 3 years ago

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress and in documentation i could not find how setup sasl/ssl

Hubbitus commented 3 years ago

Sorry, maybe you may at least extend logging what happened and what wrong?

jbvmio commented 3 years ago

i have the same issue

⇒  kafkactl -B 15.247.193.149:32764 --verbose
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] Initializing new client
INFO[2020-10-05T19:01:53+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:53+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:53+03:00] Closed connection to broker 15.247.193.149:32764
INFO[2020-10-05T19:01:53+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:53+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:53+03:00] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[2020-10-05T19:01:54+03:00] client/metadata fetching metadata for all topics from broker 15.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] Connected to broker at 15.247.193.149:32764 (unregistered)
INFO[2020-10-05T19:01:54+03:00] client/metadata got error from broker -1 while fetching metadata: EOF
INFO[2020-10-05T19:01:54+03:00] Closed connection to broker 185.247.193.149:32764
INFO[2020-10-05T19:01:54+03:00] client/metadata no available broker to send metadata request to
INFO[2020-10-05T19:01:54+03:00] client/brokers resurrecting 1 dead seed brokers
INFO[2020-10-05T19:01:54+03:00] Closing Client
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

we have a kafka at k8s cluster and exposed via ingress and in documentation i could not find how setup sasl/ssl

@zotona You can configure sasl/ssl using the following example in your config file:

  devel-cluster:
    name: devel-cluster
    brokers:
    - devel01.example.com:9092
    - devel02.example.com:9092
    - devel03.example.com:9092
    zookeeper:
    - devel01.example.com:2181
    - devel02.example.com:2181
    - devel03.example.com:2181
    clientVersion: 2.1.1
    ssl:
      insecure: true
      tlscert: /path/to/cert.pem
      tlsca: /path/to/ca.pem
      tlskey: /path/to/key.pem
    sasl:
      username: USER
      password: PASS
jbvmio commented 3 years ago

@Hubbitus

It seems there is either a connectivity issue or more likely the version of Kafka your brokers are running are not compatible. Did you ever confirm the version of Kafka your brokers are running?

jbvmio commented 3 years ago

@Hubbitus - As a test, try the forcing the following client versions:

--version 0.10.1.0
--version 0.10.2.0
--version 0.11.0.0
Hubbitus commented 3 years ago 
$ kafkactl --verbose get topics                                                                                                                   
INFO[2021-03-26T17:31:36+03:00] Failed to connect to broker ecsc00a060af.epam.com:9092: dial tcp: i/o timeout 
Error connecting to Kafka: No Connectivity to Provided Brokers
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.1.0 get topics
INFO[2021-03-26T17:31:50+03:00] Initializing new client                      
INFO[2021-03-26T17:31:50+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:31:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42192->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:31:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:31:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:31:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:31:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:31:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42242->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.10.2.0 get topics
INFO[2021-03-26T17:32:15+03:00] Initializing new client                      
INFO[2021-03-26T17:32:15+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:15+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:20+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42404->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:20+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:20+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:20+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:20+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:20+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:25+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42446->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:25+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:25+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:25+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:25+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 0.11.0.0 get topics
INFO[2021-03-26T17:32:35+03:00] Initializing new client                      
INFO[2021-03-26T17:32:35+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:35+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:40+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42570->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:40+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:40+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:40+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:40+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:41+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:41+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:46+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42612->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:46+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:46+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:46+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:46+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
[pasha@hubbitus kafkactl]$ 
[pasha@hubbitus kafkactl]$ kafkactl --verbose --version 1.0.0 get topics
INFO[2021-03-26T17:32:51+03:00] Initializing new client                      
INFO[2021-03-26T17:32:51+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:51+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:32:56+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42694->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:32:56+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:32:56+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:32:56+03:00] client/metadata retrying after 250ms... (1 attempts remaining) 
INFO[2021-03-26T17:32:56+03:00] client/metadata fetching metadata for all topics from broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:32:56+03:00] Connected to broker at ecsc00a060af.epam.com:9092 (unregistered) 
INFO[2021-03-26T17:33:01+03:00] client/metadata got error from broker -1 while fetching metadata: read tcp 100.64.60.142:42736->10.6.194.60:9092: i/o timeout 
INFO[2021-03-26T17:33:01+03:00] Closed connection to broker ecsc00a060af.epam.com:9092 
INFO[2021-03-26T17:33:01+03:00] client/metadata no available broker to send metadata request to 
INFO[2021-03-26T17:33:01+03:00] client/brokers resurrecting 1 dead seed brokers 
INFO[2021-03-26T17:33:01+03:00] Closing Client                               
Error: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

On server: Confluent 5.3.2, Kafka 2.3.x

And off course such cluster reachable from kafkacat and confluent utilities.