AWS MSK Kafka migration -- error connecting to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?

johnson101119 commented 2 years ago

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go

"github.com/Shopify/sarama" --> I am not sure with the version below version copied from config config.Version = sarama.V0_10_0_0 | kafka_2.12-2.2.1

Configuration

What configuration values are you using for Sarama and Kafka?

        config.Producer.RequiredAcks = sarama.WaitForAll
        config.Producer.Retry.Max = 1
        config.Producer.Return.Successes = true
        config.Version = sarama.V0_10_0_0
        //  config.Producer.Compression = sarama.CompressionSnappy
        Producer, Error = sarama.NewSyncProducer(configStore.ConfigStruct.GetTomlInstance().Kafka.KafkaBrokerList, config)

AWS broker endpoint TLS kafkaBrokerList = ["b-2.test-kafka-dev.kafka.us-west-2.amazonaws.com:9094","b-1.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094","b-3.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094"]

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

2021/12/09 14:17:16 kafka.go:74: Attempting to connect to Kafka 2021/12/09 14:17:16 kafka.go:85: error connecting to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Problem Description

On trying to attempt Kafka connect --> AWS MSK from ECS docker image noticed following error below.

Error 2021/12/06 16:58:04 kafka.go:85: error connecting to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Kafka topic is successfully created bin/kafka-topics.sh --describe --topic session_info --zookeeper z-3.test-kafka-dev.6sn6tf.c4.kafka.us-west-2.amazonaws.com:2181..... OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Topic:wisp_session_info PartitionCount:3 ReplicationFactor:3 Configs: Topic: session_info Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: session_info Partition: 1 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1 Topic: session_info Partition: 2 Leader: 2 Replicas: 2,1,3 Isr: 2,1,3

The broker endpoint configured is kafkaBrokerList = ["b-2.test-kafka-dev.kafka.us-west-2.amazonaws.com:9094","b-1.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094","b-3.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094"]

To validate the broker endpoint, I tested with sample docker image and able to connect to AWS broker endpoint and publish the event from the container.

Note: At on premise kafka broker connection string is configured with IP Address and works fine, Where as in AWS it is b-2.test-kafka-dev.kafka.us-west-2.amazonaws.com:9094","b-1.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094","b-3.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094 and throws an error as above.

Any help will be appreciated.

Thanks & Regards Johnson

liangxj8 commented 2 years ago

I think you can add sarama.Logger = log.New(os.Stdout, "[sarama] ", log.LstdFlags) at the beginning of your code ? I've got the log like client/metadata got error from broker -1 while fetching metadata: dial tcp: lookup b-1.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com: no such host. And when I tried nslookup, I didn't get the resolution. But maybe it is the problem of my network. FYI.

johnson101119 commented 2 years ago

will update the log file asap. Meanwhile replaced aws broker endpoint to broker VPC ipaddress but still same error.

From kafkaBrokerList = ["b-2.test-kafka-dev.kafka.us-west-2.amazonaws.com:9094","b-1.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094","b-3.test-kafka-dev.c4.kafka.us-west-2.amazonaws.com:9094"] To kafkaBrokerList = ["10.0.1.1:9094", "10.0.12.1:9094","10.0.15.1:9094] //The IP address are the sample ones where as port exact one and is common

Error 2021/12/09 14:17:16 kafka.go:74: Attempting to connect to Kafka 2021/12/09 14:17:16 kafka.go:85: error connecting to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

johnson101119 commented 2 years ago

Here is the output of Sarama log, I created the new Cluster without TLS and to have only plain text.

021/12/18 03:06:01 Attempting to connect to Kafka 2021/12/18 03:06:01 Kafka broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092

sarama] 2021/12/18 03:06:01 Initializing new client [sarama] 2021/12/18 03:06:01 ClientID is the default of 'sarama', you should consider setting it to something application-specific. [sarama] 2021/12/18 03:06:01 ClientID is the default of 'sarama', you should consider setting it to something application-specific. [sarama] 2021/12/18 03:06:01 client/metadata fetching metadata for all topics from broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092 [sarama] 2021/12/18 03:06:01 Failed to connect to broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: dial tcp: address b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: too many colons in address

[sarama] 2021/12/18 03:06:01 client/metadata got error from broker -1 while fetching metadata: dial tcp: address b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: too many colons in address [sarama] 2021/12/18 03:06:01 client/metadata no available broker to send metadata request to [sarama] 2021/12/18 03:06:01 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/12/18 03:06:01 client/metadata retrying after 250ms... (3 attempts remaining) [sarama] 2021/12/18 03:06:02 ClientID is the default of 'sarama', you should consider setting it to something application-specific. [sarama] 2021/12/18 03:06:02 client/metadata fetching metadata for all topics from broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092 [sarama] 2021/12/18 03:06:02 Failed to connect to broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: dial tcp: address b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: too many colons in address [sarama] 2021/12/18 03:06:02 client/metadata got error from broker -1 while fetching metadata: dial tcp: address b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092: too many colons in address [sarama] 2021/12/18 03:06:02 client/metadata no available broker to send metadata request to [sarama] 2021/12/18 03:06:02 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/12/18 03:06:02 client/metadata retrying after 250ms... (2 attempts remaining) [sarama] 2021/12/18 03:06:02 ClientID is the default of 'sarama', you should consider setting it to something application-specific. [sarama] 2021/12/18 03:06:02 client/metadata fetching metadata for all topics from broker b-2.notls-kafka.us-west-2.amazonaws.com:9092,b-3.notls-kafka.us-west-2.amazonaws.com:9092,b-1.notls-kafka.us-west-2.amazonaws.com:9092

..... .... sarama] 2021/12/18 03:06:02 client/metadata no available broker to send metadata request to [sarama] 2021/12/18 03:06:02 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/12/18 03:06:02 Closing Client

with IP address too got same result [sarama] 2021/12/18 03:19:21 Failed to connect to broker 10.6.1.1:9092,10.6.4.3:9092,10.6.4.10:9092: dial tcp: address 10.6.1.1:9092,10.6.4.3:9092,10.6.4.10:9092: too many colons in address

With only one IP , it is timeout error

Trying few more options, too parse the connection string,

dnwe commented 2 years ago

In this most recent log it looks like you’ve put the comma separated three broker addresses in the config as a single string rather than as a slice of strings as the Dial is setting all three and refusing to permit more than one colon (:)

johnson101119 commented 2 years ago

I could able to resolve the connectivity issue after slice of a strings for Plaintext AWS MSK Kafka Connectivity.

However for TLS based Kafka connectivity for GoLang with Sarama package did not find the relevant documentation, I referred following link below for the steps but not success on creating the server-cer.pem, client-cer.pem and client-key.pem using keytool. https://www.process-one.net/blog/using-tls-authentication-for-your-go-kafka-client/

michelb-ciandt commented 2 years ago

Hi @johnson101119, it seems we have the same issue with AWS MSK with TLS, in my case I'm trying to use it with the AWS EKS cluster.

check my raised discussion #2636, you can check my configs files.

Are you trying to use Keda trigger auth Kafka credential file Could you please share your configs? ScaledObject, etc...

and also please tell me what are the versions of Keda and AWS MSK

Constantin07 commented 2 years ago

I'm having similar issues with AWS MSK with TLS_PLAINTEXT when in CA is used not just a single Root CA but a chain of Root & Intermediate CAs.

│ Error: kafka: client has run out of available brokers to talk to: 3 errors occurred:
│   * local error: tls: unexpected message
│   * local error: tls: unexpected message
│   * local error: tls: unexpected message

I'm using Terraform Provider for Kafka which is based on this library, see the detailed description here https://github.com/Mongey/terraform-provider-kafka/issues/272

AWS MSK version 2.4.1.1 Shopify/sarama version v1.34.1

Please note that the Kafka tools like kafka-acls.sh, kafka-topics.sh and others work just fine when same credentials (client key, cert, CA chain) are used in PKSC12 keystore.

Constantin07 commented 2 years ago

Here is the log I get with sarama client configured with TLS:

[sarama] 2022/06/28 10:44:55 Initializing new client
[sarama] 2022/06/28 10:44:55 Successfully initialized new client
[sarama] 2022/06/28 10:44:55 client/metadata fetching metadata for [test-topic] from broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 Connected to broker at b-1.*******.eu-west-1.amazonaws.com:9094 (unregistered)
[sarama] 2022/06/28 10:44:55 client/metadata got error from broker -1 while fetching metadata: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Closed connection to broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 client/metadata no available broker to send metadata request to
[sarama] 2022/06/28 10:44:55 Error while sending ApiVersionsRequest to broker b-1.*******.eu-west-1.amazonaws.com:9094: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 client/brokers resurrecting 1 dead seed brokers
[sarama] 2022/06/28 10:44:55 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2022/06/28 10:44:55 client/metadata fetching metadata for [test-topic] from broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 Connected to broker at b-1.*******.eu-west-1.amazonaws.com:9094 (unregistered)
[sarama] 2022/06/28 10:44:55 client/metadata got error from broker -1 while fetching metadata: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Error while sending ApiVersionsRequest to broker b-1.*******.eu-west-1.amazonaws.com:9094: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Closed connection to broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 client/metadata no available broker to send metadata request to
[sarama] 2022/06/28 10:44:55 client/brokers resurrecting 1 dead seed brokers
[sarama] 2022/06/28 10:44:55 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2022/06/28 10:44:55 client/metadata fetching metadata for [test-topic] from broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 Connected to broker at b-1.*******.eu-west-1.amazonaws.com:9094 (unregistered)
[sarama] 2022/06/28 10:44:55 client/metadata got error from broker -1 while fetching metadata: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Closed connection to broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 client/metadata no available broker to send metadata request to
[sarama] 2022/06/28 10:44:55 Error while sending ApiVersionsRequest to broker b-1.*******.eu-west-1.amazonaws.com:9094: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 client/brokers resurrecting 1 dead seed brokers
[sarama] 2022/06/28 10:44:55 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2022/06/28 10:44:55 client/metadata fetching metadata for [test-topic] from broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 Connected to broker at b-1.*******.eu-west-1.amazonaws.com:9094 (unregistered)
[sarama] 2022/06/28 10:44:55 client/metadata got error from broker -1 while fetching metadata: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Error while sending ApiVersionsRequest to broker b-1.*******.eu-west-1.amazonaws.com:9094: local error: tls: unexpected message
[sarama] 2022/06/28 10:44:55 Closed connection to broker b-1.*******.eu-west-1.amazonaws.com:9094
[sarama] 2022/06/28 10:44:55 client/metadata no available broker to send metadata request to
[sarama] 2022/06/28 10:44:55 client/brokers resurrecting 1 dead seed brokers
2022/06/28 10:44:55 Unable to fetch partition IDs for the topic test-topic kafka: client has run out of available brokers to talk to: local error: tls: unexpected message

binbinshi commented 1 year ago

see the same error kafka: client has run out of available brokers to talk to (Is your cluster reachable?) i add sarama config version ， the error disappeared config := sarama.NewConfig() config.Version = sarama.V0_11_0_0

github-actions[bot] commented 1 year ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

dnwe commented 1 year ago

From the linked issue it looks like TLS connectivity to AWS MSK was fixed by including the full chain in the client cert pem for tls client authentication

Sorted out by appending the Intermediate CA file to the client_cert .

Deeptanshu-sankhwar commented 1 week ago

see the same error kafka: client has run out of available brokers to talk to (Is your cluster reachable?) i add sarama config version ， the error disappeared config := sarama.NewConfig() config.Version = sarama.V0_11_0_0

this worked for me! not entirely sure, why

IBM / sarama