PrabhanjanDesai commented 4 months ago

Description

We have configured strimzi kafka with 9092 as non secured port and with 9093 as secured port. While trying to produce the messages by connecting to 9092 (non secured port) "SASL Authentication error" is thrown intermittently. This error should be thrown while connecting to secure port (9093) when there is an authentication issue.

Error details - Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-38, started daemon 139807049099008)>> Traceback (most recent call last): File "/kafka-wrapper/producer_oauth.py", line 87, in produce_to_topic producer.poll(0.1) File "/kafka-wrapper/producer_oauth.py", line 25, in error_callback raise(err) cimpl.KafkaError: KafkaError{code=_TRANSPORT,val=-195,str="kafka-bootstraphost-0.kafka-bootstraphost-brokers.namespace.svc:9092/0: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1ms in state UP)"} During handling of the above exception, another exception occurred:

How to reproduce

NA

Checklist

Please provide the following information:

[ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.3.0
[ ] Apache Kafka broker version:
[ ] Client configuration: {...} -

config = {

| 'bootstrap.servers': self.broker, | 'linger.ms': 60, | 'queue.buffering.max.messages': 5000000, | 'compression.type': 'lz4', | 'batch.num.messages': 50000, | 'batch.size': 200000, | 'logger': logger, | 'acks': 0 , | 'error_cb': self.error_callback, | }

[ ] Operating system: SUSE linux
[ ] Provide client logs (with 'debug': '..' as necessary) - Unhandled exception in thread started by <bound method Thread._bootstrap of <Thread(Thread-38, started daemon 139807049099008)>> Traceback (most recent call last): File "/kafka-wrapper/producer_oauth.py", line 87, in produce_to_topic producer.poll(0.1) File "/kafka-wrapper/producer_oauth.py", line 25, in error_callback raise(err) cimpl.KafkaError: KafkaError{code=_TRANSPORT,val=-195,str="kafka-bootstraphost-0.kafka-bootstraphost-brokers.namespace.svc:9092/0: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 1ms in state UP)"} During handling of the above exception, another exception occurred:
[ ] Provide broker log excerpts - Not Available
[ ] Critical issue - Not Available

pranavrth commented 3 months ago

By secured and non secured, you mean authentication (SASL) or secured communication (SSL)?

What do you mean by intermittent? Does it work properly for some time and then disconnects randomly?

In general, this error happens when the broker listener is configured for SASL authentication but the client is not. Can you please verify this again once?

PrabhanjanDesai commented 3 months ago

By secured and non secured, you mean authentication (SASL) or secured communication (SSL)? It's MTLS SSL secure communication What do you mean by intermittent? Does it work properly for some time and then disconnects randomly? Yes, correct In general, this error happens when the broker listener is configured for SASL authentication but the client is not. Can you please verify this again once? **We are connecting to insecure port , below is our kafka configuration : kafka: authorization: superUsers:

ANONYMOUS type: simple config: auto.create.topics.enable: false default.replication.factor: 3 inter.broker.protocol.version: "3.3" min.insync.replicas: 2 offsets.topic.replication.factor: 3 retries: 10 retry.backoff.ms: 2000 transaction.state.log.min.isr: 2 transaction.state.log.replication.factor: 3 listeners:

name: plain port: 9092 tls: false type: internal name: external port: 9094 tls: true type: cluster-ip**

pranavrth commented 3 months ago

Can you please enable debug logging and provide the generated logs?

adrian0ribeir0 commented 3 months ago

Hi guys

I'm having same problem to connect to aws msk sasl_cram,

Producer config I tried with version 2.0.2, 2,2.0 and 2.3.0

    producer_conf = {
        'bootstrap.servers': boostrap_server,
        "client.id": f"efop-sap-integration-admin-{uuid.uuid4().hex}",
        "acks": "all",
        "enable.idempotence": True,
        "security.protocol": "SASL_SSL",
        "sasl.mechanism": "SCRAM-SHA-512",
        "sasl.username": config.sasl_username,
        "sasl.password": config.sasl_password,
    }

    return Producer(producer_conf)

I just remove the server identification form url logs

│ %7|1711465645.879|TOPPAR|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://amazonaws.com:9096/2: extraction_hubspot_cha │
│ %7|1711465645.879|PRODUCE|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_ch │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://.amazonaws.com:9096/2: Sent partial ProduceRequ │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://.amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Sent partial ProduceRequ │
│ %7|1711465645.879|SEND|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Sent ProduceRequest (v7, │
│ %7|1711465645.883|RECV|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: Received ProduceResponse │
│ %7|1711465645.883|MSGSET|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_cha │
│ 200 POST /v1/api/driverhubspot/raw/change-tracking (10.42.3.52) 1069.67ms                                                                                                                                                                                                    │
│ %6|1711465647.034|FAIL|webhook-deal-changetracking-d5b58b4caa6c462b9d0985fb9475d2b1#producer-3| [thrd:amazonaws.com:9096/b]: b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/bootstrap: Disconnected while requesti │
│ %7|1711465648.574|PARTITIONER|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: extraction_hubspot_changeset [5] is the new sticky partition                                                                                              │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: sasl_ssl://amazonaws.com:9096/2: Wake-up: flushing                                                                  │
│ %7|1711465648.574|TOPPAR|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:sasl_ssl://amazonaws]: sasl_ssl://b-2.efopsystems.wiz11c.c12.kafka.us-east-1.amazonaws.com:9096/2: extraction_hubspot_cha │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: sasl_ssl://amazonaws.com:9096/1: Wake-up: flushing                                                                  │
│ %7|1711465648.574|WAKEUP|webhook-deal-changetracking-20f2263f3ad3444b942df50816cd06b0#producer-1| [thrd:app]: Wake-up sent to 2 broker threads in state >= UP: flushing

confluentinc / confluent-kafka-python

SASL Authentication issue #1718

Description

How to reproduce

Checklist

config = {