search

birdayz / kaf

Modern CLI for Apache Kafka, written in Go.
Apache License 2.0
2.19k stars 142 forks source link

Cannot connect to broker #61

Open jinleileiking opened 5 years ago

jinleileiking commented 5 years ago

kaf version: v0.1.20

kafka version: kafka_2.11-0.10.0.0

[jinlei1@tjwq01-ibu-video007106:~] master(+9/-4,10)* 3s ± kaf -b 10.64.2.163:8092 topics -v
[sarama] 2019/07/18 16:56:05 client.go:127: Initializing new client
[sarama] 2019/07/18 16:56:05 config.go:494: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/07/18 16:56:05 config.go:494: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/07/18 16:56:05 client.go:775: client/metadata fetching metadata for all topics from broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:05 broker.go:214: Connected to broker at 10.64.2.163:8092 (unregistered)
[sarama] 2019/07/18 16:56:05 client.go:813: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2019/07/18 16:56:05 broker.go:253: Closed connection to broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:05 client.go:824: client/metadata no available broker to send metadata request to
[sarama] 2019/07/18 16:56:05 client.go:565: client/brokers resurrecting 1 dead seed brokers
[sarama] 2019/07/18 16:56:05 client.go:759: client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2019/07/18 16:56:05 config.go:494: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/07/18 16:56:05 client.go:775: client/metadata fetching metadata for all topics from broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:05 broker.go:214: Connected to broker at 10.64.2.163:8092 (unregistered)
[sarama] 2019/07/18 16:56:05 client.go:813: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2019/07/18 16:56:05 broker.go:253: Closed connection to broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:05 client.go:824: client/metadata no available broker to send metadata request to
[sarama] 2019/07/18 16:56:05 client.go:565: client/brokers resurrecting 1 dead seed brokers
[sarama] 2019/07/18 16:56:05 client.go:759: client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2019/07/18 16:56:06 config.go:494: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/07/18 16:56:06 client.go:775: client/metadata fetching metadata for all topics from broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:06 broker.go:214: Connected to broker at 10.64.2.163:8092 (unregistered)
[sarama] 2019/07/18 16:56:06 client.go:813: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2019/07/18 16:56:06 broker.go:253: Closed connection to broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:06 client.go:824: client/metadata no available broker to send metadata request to
[sarama] 2019/07/18 16:56:06 client.go:565: client/brokers resurrecting 1 dead seed brokers
[sarama] 2019/07/18 16:56:06 client.go:759: client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2019/07/18 16:56:06 config.go:494: ClientID is the default of 'sarama', you should consider setting it to something application-specific.
[sarama] 2019/07/18 16:56:06 client.go:775: client/metadata fetching metadata for all topics from broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:06 broker.go:214: Connected to broker at 10.64.2.163:8092 (unregistered)
[sarama] 2019/07/18 16:56:06 client.go:813: client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2019/07/18 16:56:06 broker.go:253: Closed connection to broker 10.64.2.163:8092
[sarama] 2019/07/18 16:56:06 client.go:824: client/metadata no available broker to send metadata request to
[sarama] 2019/07/18 16:56:06 client.go:565: client/brokers resurrecting 1 dead seed brokers
[sarama] 2019/07/18 16:56:06 client.go:227: Closing Client
jinleileiking commented 5 years ago

use tag v0.1.17 ok.

birdayz commented 5 years ago

can you confirm that the latest version (eg from master) works with an older version? on my side it's fine, can't tell the broker version though right now

edit: problem might be that kaf does not correctly set the broker version. it's currently hardcoded to some specific version i think. i'll investigate this evening

jinleileiking commented 5 years ago

can you confirm that the latest version (eg from master) works with an older version?

---- Can you elaborate this ?

jinleileiking commented 5 years ago

I change Version

  saramaConfig.Version = sarama.V0_10_0_0¬

The log says:

Unable to list topics: kafka server: The version of API is not supported

Seems LIstTopics do not support 0.10.0.0

birdayz commented 5 years ago

Okay now i understood, somehow i thought you use kafka version 2.x.x. but it's actually 0.10 and 2.11 is the scala version.

i think it would make sense to add either auto detection of the kafka version (kaf currently hardcodes it to 1.0.0), or making it configurable in the kaf config file.

thanks for digging into this!

jinleileiking commented 5 years ago

Yes. My kafka version is 2.11-0.10.0.0

jinleileiking commented 5 years ago

The following steps are worked for my kafka:

jinleileiking commented 5 years ago

Seems there are significant changes in the sarama lib.

jinleileiking commented 5 years ago

And sarama docs said:

// The version of Kafka that Sarama will assume it is running against.
    // Defaults to the oldest supported stable version. Since Kafka provides
    // backwards-compatibility, setting it to a version older than you have
    // will not break anything, although it may prevent you from using the
    // latest features. Setting it to a version greater than you are actually
    // running may lead to random breakage.
    Version KafkaVersion
jinleileiking commented 5 years ago

In my fork https://github.com/jinleileiking/kaf

Works well. I change the kafka version with latest code. It is very confusing.

birdayz commented 5 years ago

this shows to me that there were some changes in sarama that now make your connection fail if the wrong kafka version is set - which is actually good behavior.

so we need to make the version configurable. are you interested in opening a PR for this?

Note that some commands are not available with certain versions - that's why i default to v1.1.0.0. but from your logs i see that sarama will return this error anyway ("the version of this api is not supported").

jinleileiking commented 5 years ago

https://github.com/Shopify/sarama/issues/1431

Seems version should be 11. I do not think my fork is correct. It needs dig the sarama codes to get the truth. I didnot have time for that right now.

superChing commented 5 years ago

same here kafka_2.11-0.10 only work below v0.1.17

but v0.1.17 give error at topics listing

➜ kaf topics
panic: EOF

goroutine 1 [running]:
main.glob..func12(0x18e8920, 0x190c730, 0x0, 0x0)
    /home/j0e/gopath/src/github.com/infinimesh/kaf/cmd/kaf/topic.go:59 +0x54e
github.com/infinimesh/kaf/vendor/github.com/spf13/cobra.(*Command).execute(0x18e8920, 0x190c730, 0x0, 0x0, 0x18e8920, 0x190c730)
    /home/j0e/gopath/src/github.com/infinimesh/kaf/vendor/github.com/spf13/cobra/command.go:766 +0x2cc
github.com/infinimesh/kaf/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x18e7880, 0x12, 0x1522e90, 0x17)
    /home/j0e/gopath/src/github.com/infinimesh/kaf/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/infinimesh/kaf/vendor/github.com/spf13/cobra.(*Command).Execute(0x18e7880, 0x0, 0x18e8920)
    /home/j0e/gopath/src/github.com/infinimesh/kaf/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /home/j0e/gopath/src/github.com/infinimesh/kaf/cmd/kaf/kaf.go:60 +0x2d