search

silviucpp / erlkaf

Erlang kafka driver based on librdkafka
MIT License
84 stars 41 forks source link

Error multiple consumers #53

Closed alexarcombe closed 1 year ago

alexarcombe commented 1 year ago

Getting this error when I try to start multiple consumers while connecting to Confluent Cloud

06:16:30.023 [error] rdkafka#consumer-1 FAIL [thrd:GroupCoordinator]: GroupCoordinator: b30-pkc-2396y.us-east-1.aws.confluent.cloud:9092: Failed to resolve 'b30-pkc-2396y.us-east-1.aws.confluent.cloud:9092': nodename nor servname provided, or not known (after 19ms in state CONNECT, 4 identical error(s) suppressed) 06:16:30.024 [error] rdkafka#consumer-1 ERROR [thrd:app]: rdkafka#consumer-1: GroupCoordinator: b30-pkc-2396y.us-east-1.aws.confluent.cloud:9092: Failed to resolve 'b30-pkc-2396y.us-east-1.aws.confluent.cloud:9092': nodename nor servname provided, or not known (after 19ms in state CONNECT, 4 identical error(s) suppressed)

Everything is working fine connecting to a local Kafka server (without sasl), and connecting one consumer to Confluent Cloud is working perfectly.

Is there any configuration or something I can do to get it working with more than one consumer?

Thanks in advance.

silviucpp commented 1 year ago

This is not a bug related to library. Check your networking. The error is pretty clear:

Failed to resolve 'b30-pkc-2396y.us-east-1.aws.confluent.cloud:9092'

Check connectivity between the machine where you running erlkaf and the machine where the broker is running (b30-pkc-2396y.us-east-1.aws.confluent.cloud port 9092).

alexarcombe commented 1 year ago

Ok, but I am able to connect if I just use one consumer to the same broker, the problem is when i'm running two or more. How would I go about to resolve that?

alexarcombe commented 1 year ago

This doesn't work

config :erlkaf, global_client_options: [ bootstrap_servers: "pkc-2396y.us-east-1.aws.confluent.cloud:9092", ], clients: [ broker: [ type: :consumer, group_id: "erlkaf-broker", topics: [{"ViewerSessionEnded_v2", [callback_module: ErlkafExample.Batcher]}], topic_options: [ auto_offset_reset: :latest ], client_options: [ security_protocol: "sasl_ssl", sasl_mechanisms: "PLAIN", sasl_username: "...", sasl_password: "...", ] ], individual_submitter: [ type: :consumer, group_id: "erlkaf-individual-submitter", topics: [{"ViewerSessionEnded_v2", [callback_module: ErlkafExample.IndividualSubmitter]}], topic_options: [ auto_offset_reset: :latest ], client_options: [ security_protocol: "sasl_ssl", sasl_mechanisms: "PLAIN", sasl_username: "...", sasl_password: "...", ] ] ]

This works config :erlkaf, global_client_options: [ bootstrap_servers: "pkc-2396y.us-east-1.aws.confluent.cloud:9092", ], clients: [ broker: [ type: :consumer, group_id: "erlkaf-broker", topics: [{"ViewerSessionEnded_v2", [callback_module: ErlkafExample.Batcher]}], topic_options: [ auto_offset_reset: :latest ], client_options: [ security_protocol: "sasl_ssl", sasl_mechanisms: "PLAIN", sasl_username: "...", sasl_password: "...", ] ] ]

Hard for me to see how there could be a connection problem if one works but two doesn't 🤔

silviucpp commented 1 year ago

Hello,

We are using over 20 consumers on the same topic processing over 100 M messages every day.

To receive help please format your config properly to be able to follow your config and provide more information regarding the number of partitions you have broker version etc.