How consumer can use proxy setting to communicate to cluster

vjakka-intel commented 2 years ago

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go		v.37.1	---------	yes

Configuration

   config.ClientID = pid                              
config.Net.SASL.Mechanism = sarama.SASLTypePlaintext //"PLAIN"
config.Net.SASL.Enable = true
config.Net.SASL.User = "xxxxxxxxxxxxxxx"
config.Net.SASL.Password = "asAAAAAAAAAAAAAAAA
config.Net.TLS.Enable = true
tlsConfig := &tls.Config{
    InsecureSkipVerify: true,
    ClientAuth:         0,
}
config.Net.TLS.Config = tlsConfig
config.Producer.Retry.Max = 5
config.Producer.RequiredAcks = sarama.WaitForAll
brokers := []string{"abcd.asad.c2.kafka.us-west-1.amazonaws.com:9092"}
config.Producer.Return.Successes = true
config.Consumer.Offsets.Initial = sarama.OffsetNewest

What configuration values are you using for Sarama and Kafka?

Logs

When filing an issue please provide logs from Sarama and Kafka if at all possible. You can set sarama.Logger to a log.Logger to capture Sarama debug output.

logs: CLICK ME

[sarama] 2022/10/14 23:11:52 client/metadata fetching metadata for all topics from broker b-2.vjmskkafkaprisubnet.qbh0ve.c2.kafka.us-west-1.amazonaws.com:9092 [sarama] 2022/10/14 23:11:52 Failed to send SASL handshake abcd.asad.c2.kafka.us-west-1.amazonaws.com:9092: EOF [sarama] 2022/10/14 23:11:52 Closed connection to broker abcd.asad.c2.kafka.us-west-1.amazonaws.com:9092 [sarama] 2022/10/14 23:11:52 client/metadata got error from broker -1 while fetching metadata: EOF [sarama] 2022/10/14 23:11:52 client/metadata fetching metadata for all topics from brokerabcd.asad.c2.kafka.us-west-1.amazonaws.com:9092 [sarama] 2022/10/14 23:11:52 Failed to send SASL handshake b-1.vjmskkafkaprisubnet.qbh0ve.c2.kafka.us-west-1.amazonaws.com:9092: EOF

``` ```

Problem Description

Since my microservice is running in corporate firewall, we are setting proxy parameters.

My question is, how do we need to specify in consumer properties to use proxy while consuming the message ? Any new configuration parameters need to pass to sarama config library ?

How do we specify for producer to use proxy to communicate with kafka cluster ?

Firewall rule is allowed to communicate with all 9xxx ports.

Any suggestion please ?

vjakka-intel commented 1 year ago

resolved

lawrenz1 commented 1 year ago

what was your solution to this?

IBM / sarama