vikramindian commented 4 years ago

Description

There is a topic in secured kafka cluster on which I don't have write and read permissions. I started a producer and a consumer on this topic. My producer did not send any messages and even did not get any thing in delivery callback.

In debug logs, I could see 'Broker: Topic authorization failed' messages

%7|1575552397.201|METADATA|testprod#producer-1| [thrd:main]: Error in metadata reply for topic test-33 (PartCnt 0): Broker: Topic authorization failed
%7|1575552397.201|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: 1/1 requested topic(s) seen in metadata
%7|1575552397.201|CLUSTERID|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: ClusterId update "" -> "52dx8qR5T1KKN8UT92peWA"
%7|1575552397.201|CONTROLLERID|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: ControllerId update -1 -> 1
%7|1575552397.201|METADATA|testprod#producer-1| [thrd:main]: Purged 1/1 cached topic hint(s)
%7|1575552397.201|BROADCAST|testprod#producer-1| [thrd:main]: Broadcasting state change
%7|1575552397.201|BRKMAIN|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: Enter main broker thread
%7|1575552397.998|NOINFO|testprod#producer-1| [thrd:main]: Topic test-33 metadata information unknown
%7|1575552397.998|NOINFO|testprod#producer-1| [thrd:main]: Topic test-33 partition count is zero: should refresh metadata
%7|1575552397.998|METADATA|testprod#producer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1575552397.998|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: Request metadata for 1 topic(s): refresh unavailable topics
%7|1575552397.998|SEND|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: Sent MetadataRequest (v2, 35 bytes @ 0, CorrId 7)
%7|1575552397.999|RECV|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: Received MetadataResponse (v2, 275 bytes, CorrId 7, rtt 0.56ms)
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: ClusterId: 52dx8qR5T1KKN8UT92peWA, ControllerId: 1
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap: 5 brokers, 1 topics
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Broker #0/5: mwkafka-staging-01.tbd.com:9092 NodeId 5
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Broker #1/5: mwkafka-staging-02.dr.com:9092 NodeId 4
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Broker #2/5: mwkafka-staging-01.nyc.com:9092 NodeId 1
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Broker #3/5: mwkafka-staging-02.nyc.com:9092 NodeId 2
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Broker #4/5: mwkafka-staging-01.dr.com:9092 NodeId 3
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Topic #0/1: test-33 with 0 partitions: Broker: Topic authorization failed
%7|1575552397.999|METADATA|testprod#producer-1| [thrd:main]: Error in metadata reply for topic test-33 (PartCnt 0): Broker: Topic authorization failed

Later I repeated same thing after I got only Read permissions on that topic. This time the observed debug messages were different and I got 'Topic authorization failed' message in delivery call back this time.

Current ACLs for resource `Topic:LITERAL:test-33`: 
    User:vikram has Allow permission for operations: Read from hosts: *

%7|1575984353.864|METADATA|testprod#producer-1| [thrd:main]: sasl_plaintext://mwkafka-staging-01.dr:9092/bootstrap:   Topic #0/1: test-33 with 3 partitions
%7|1575984353.864|STATE|testprod#producer-1| [thrd:main]: Topic test-33 changed state unknown -> exists
%7|1575984353.864|PARTCNT|testprod#producer-1| [thrd:main]: Topic test-33 partition count changed from 0 to 3
%7|1575984353.864|BRKMAIN|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: Enter main broker thread
%7|1575984353.864|TOPPARNEW|testprod#producer-1| [thrd:main]: NEW test-33 [0] 0x7fb7f4007d80 (at rd_kafka_topic_partition_cnt_update:620)
%7|1575984353.864|TOPPARNEW|testprod#producer-1| [thrd:main]: NEW test-33 [1] 0x7fb7f4008280 (at rd_kafka_topic_partition_cnt_update:620)
%7|1575984353.864|TOPPARNEW|testprod#producer-1| [thrd:main]: NEW test-33 [2] 0x7fb7f4008780 (at rd_kafka_topic_partition_cnt_update:620)

%7|1575984354.158|SEND|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: Sent ProduceRequest (v7, 140 bytes @ 0, CorrId 6)
%7|1575984354.158|RECV|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: Received ProduceResponse (v7, 51 bytes, CorrId 6, rtt 0.34ms)
%7|1575984354.158|REQERR|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: ProduceRequest failed: Broker: Topic authorization failed: actions Permanent,MsgNotPersisted
%7|1575984354.158|MSGSET|testprod#producer-1| [thrd:sasl_plaintext://mwkafka-staging-01.dr.com:9092/3]: sasl_plaintext://mwkafka-staging-01.dr.com:9092/3: test-33 [2]: MessageSet with 1 message(s) (MsgId 0, BaseSeq -1) encountered error: Broker: Topic authorization failed (actions Permanent,MsgNotPersisted)

What did I figure out from this? In first case since I did not have Read permissions, producer failed to even fetch Topic Metadata and so it did not proceed further. Hence there are no produce requests and nothing in callbacks.

In second case, I have only Read permissions, so my producer fetched topic metadata and sent produce request but got error response and so I received this in my delivery callback.

Question

How to get authorization failed message in callback. Are there any callbacks for this?. This is because a user who is trying to send messages may not have read permissions and in such a situation he may not get to know about this.

How to reproduce

Checklist

Please provide the following information:

[ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
[ ] Apache Kafka broker version:
[ ] Client configuration: {...}
[ ] Operating system:
[ ] Provide client logs (with 'debug': '..' as necessary)
[ ] Provide broker log excerpts
[ ] Critical issue

rnpridgeon commented 4 years ago

Try registering an error callback with the client and check for authorization errors. I included a link to one of the tests as an example.

test https://github.com/confluentinc/confluent-kafka-python/blob/d0496eb9db8682a3e0288902b5569c6d5e7ecab8/tests/test_KafkaError.py#L10-L15

errors https://docs.confluent.io/current/clients/confluent-kafka-python/#kafkaerror

vikramindian commented 4 years ago

Thank you for thee response @rnpridgeon

registered error callback but it did not get any error

edenhill commented 4 years ago

You will need to call poll(), flush() or consume() to trigger the callbacks

vikramindian commented 4 years ago

@edenhill I'm calling poll() after every sending every message

rnpridgeon commented 4 years ago

Authorization is handled by the broker itself and not the client so I can't speak to the asymmetric handling of the metadata request between consumer and producer. I suspect user vikram has DESCRIBE on topic-33 in addition to read in the consumer test. Either that or READ on TOPIC provides implicit access to DESCRIBE. If the latter is true you'll want to take that up with the Kafka project itself.

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1091

With that said I would expect to see a global error propagation of the authorization error. A delivery report error does feel more appropriate though I agree. In your produce example. I'll try to reproduce this inhouse. I'll follow up with my findings afterwards.

vikramindian commented 4 years ago

I did not provide describe permissions for user vikram on topic-33 explicitly.

bpradeep20 commented 4 years ago

I am also facing the same issue https://github.com/confluentinc/confluent-kafka-python/issues/704. Can somebody please help?

azhurbilo commented 4 years ago

Any news? Does someone find solution how handle AuthorizationException when principal don't have ACL permissions to topic?

edenhill commented 4 years ago

@azhurbilo What client version are you on? Are you using producer or consumer?

azhurbilo commented 4 years ago

@edenhill producer / confluent-kafka 1.5.0

deeTEEcee commented 3 years ago

If it helps, I noticed that if you don't have the "Create TOPIC" permissions, it'll just hang. FYI, my auto.create.topics.enable setting is on. Even if you add a timeout to flush, it'll act as if no error happened.

After I added that permission (but still don't have "Write TOPIC"), I do receive callback errors saying "Failed to deliver message" as expected.

edenhill commented 3 years ago

Please repro on v1.6.0 when it is out

confluentinc / confluent-kafka-python

Not receiving any authorization error when user doesn't have write permissions #738

Description

How to reproduce

Checklist