search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
206 stars 3.14k forks source link

"Topic test partition count is zero: should refresh metadata" with SSL #2733

Closed theidexisted closed 4 years ago

theidexisted commented 4 years ago

Description

The certificate file, trust store an key store is generated by command:

gen-ssl-certs.sh ca ca-cert cert
gen-ssl-certs.sh -k server ca-cert broker_0 broker
gen-ssl-certs.sh client ca-cert client_0 client
gen-ssl-certs.sh -k client ca-cert client_0 client

In the test, we only have one Zookeeper node, one Kafka node.

Configuration for Kafka server:

#authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
auto.create.topics.enable=true
broker.id=0
listeners=SASL_SSL://10.23.4.180:9092
advertised.listeners=SASL_SSL://10.23.4.180:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
#inter.broker.protocol.version=0.10.1

advertised.host.name=10.23.4.180
num.partitions=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
#transaction.state.log.min.isr=1
#num.recovery.threads.per.data.dir=1
log.flush.interval.messages=30000000
log.flush.interval.ms=18000000
log.retention.minutes=3000
log.segment.bytes=1073741824
log.retention.check.interval.ms=3000000
delete.topic.enable=true
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
#super.users=User:admin
default.replication.factor=1
ssl.endpoint.identification.algorithm=

# SSL
ssl.protocol = TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type = JKS
ssl.keystore.location = /work_space/open-source/librdkafka-1.2.1/tests/broker_0server.keystore.jks
ssl.keystore.password = abcdefgh
ssl.key.password = abcdefgh
ssl.truststore.type = JKS
ssl.truststore.location = /work_space/open-source/librdkafka-1.2.1/tests/broker_0server.truststore.jks
ssl.truststore.password = abcdefgh
# To require authentication of clients use "require", else "none" or "request"
ssl.client.auth = required

# ACLs
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin

JAAS configuration for Kafka server:

KafkaServer {
   org.apache.kafka.common.security.plain.PlainLoginModule required
   username="admin"
   password="admin-secret"
   user_admin="admin-secret"
   user_client="client-secret";
};

Client {
   org.apache.zookeeper.server.auth.DigestLoginModule required
   username="kafka"
   password="kafka-secret";
};

Log get from rdkafka example:

1582790658.998 RDKAFKA-7-SASL: rdkafka#producer-1: [thrd:app]: Selected provider PLAIN (builtin) for SASL mechanism PLAIN
1582790658.998 RDKAFKA-7-OPENSSL: rdkafka#producer-1: [thrd:app]: librdkafka built with OpenSSL version 0x1000105f
1582790658.999 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading CA certificate(s) from file ../tests/ca-cert
1582790658.999 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading public key from file ../tests/client_0client.pem
1582790658.999 RDKAFKA-7-SSL: rdkafka#producer-1: [thrd:app]: Loading private key file from ../tests/client_0client.key
1582790658.999 RDKAFKA-7-SSLPASSWD: rdkafka#producer-1: [thrd:app]: Private key requires password
1582790658.999 RDKAFKA-7-INIT: rdkafka#producer-1: [thrd:app]: librdkafka v1.2.1-O0 (0x10201ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL SASL_CYRUS HDRHISTOGRAM SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0xffff)
1582790658.999 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd::0/internal]: :0/internal: Enter main broker thread
1582790658.999 RDKAFKA-7-WAKEUPFD: rdkafka#producer-1: [thrd:app]: sasl_ssl://10.24.3.180:9092/bootstrap: Enabled low-latency ops queue wake-ups
1582790659.000 RDKAFKA-7-BROKER: rdkafka#producer-1: [thrd:app]: sasl_ssl://10.24.3.180:9092/bootstrap: Added new broker with NodeId -1
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: sasl_ssl://10.24.3.180:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
1582790659.000 RDKAFKA-7-BRKMAIN: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Enter main broker thread
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Received CONNECT op
1582790659.000 RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
1582790659.000 RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: Broadcasting state change
1582790659.000 RDKAFKA-7-TOPIC: rdkafka#producer-1: [thrd:app]: New local topic: test
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: broker in state TRY_CONNECT connecting
1582790659.000 RDKAFKA-7-TOPPARNEW: rdkafka#producer-1: [thrd:app]: NEW test [-1] 0x10910d0 (at rd_kafka_topic_new0:393)
1582790659.000 RDKAFKA-7-STATE: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
1582790659.000 RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: Broadcasting state change
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: leader query
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: leader query
1582790659.000 RDKAFKA-7-METADATA: rdkafka#producer-1: [thrd:app]: Skipping metadata refresh of 1 topic(s): no usable brokers
1582790659.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Connecting to ipv4#10.24.3.180:9092 (sasl_ssl) with socket 7
1582790659.999 RDKAFKA-7-NOINFO: rdkafka#producer-1: [thrd:main]: Topic test metadata information unknown
1582790659.999 RDKAFKA-7-NOINFO: rdkafka#producer-1: [thrd:main]: Topic test partition count is zero: should refresh metadata
1582790659.999 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
1582790659.999 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
1582790659.999 RDKAFKA-7-METADATA: rdkafka#producer-1: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
1582790659.999 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
1582790660.999 RDKAFKA-7-NOINFO: rdkafka#producer-1: [thrd:main]: Topic test metadata information unknown
1582790661.000 RDKAFKA-7-NOINFO: rdkafka#producer-1: [thrd:main]: Topic test partition count is zero: should refresh metadata
1582790661.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: refresh unavailable topics
1582790661.000 RDKAFKA-7-CONNECT: rdkafka#producer-1: [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
1582790661.000 RDKAFKA-7-METADATA: rdkafka#producer-1: [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
1582790664.446 RDKAFKA-7-DESTROY: rdkafka#producer-1: [thrd:app]: Terminating instance (destroy flags none (0x0))
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:app]: Interrupting timers
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:app]: Sending TERMINATE to internal main thread
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:app]: Sending thread kill signal 29
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:app]: Joining internal main thread
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:main]: Internal main thread terminating
1582790664.446 RDKAFKA-7-DESTROY: rdkafka#producer-1: [thrd:main]: Destroy internal
1582790664.446 RDKAFKA-7-BROADCAST: rdkafka#producer-1: [thrd:main]: Broadcasting state change
1582790664.446 RDKAFKA-7-DESTROY: rdkafka#producer-1: [thrd:main]: Removing all topics
1582790664.446 RDKAFKA-7-TOPPARREMOVE: rdkafka#producer-1: [thrd:main]: Removing toppar test [-1] 0x10910d0
1582790664.446 RDKAFKA-7-DESTROY: rdkafka#producer-1: [thrd:main]: test [-1]: 0x10910d0 DESTROY_FINAL
1582790664.446 RDKAFKA-7-DESTROY: rdkafka#producer-1: [thrd:main]: Sending TERMINATE to sasl_ssl://10.24.3.180:9092/bootstrap
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:main]: Purging reply queue
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:main]: Decommissioning internal broker
1582790664.446 RDKAFKA-7-TERMINATE: rdkafka#producer-1: [thrd:main]: Join 2 broker thread(s)
1582790664.446 RDKAFKA-7-TERM: rdkafka#producer-1: [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
1582790664.446 RDKAFKA-7-TERM: rdkafka#producer-1: [thrd:sasl_ssl://10.24.3.180:9092/bootstrap]: sasl_ssl://10.24.3.180:9092/bootstrap: Received TERMINATE op in state CONNECT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs

How to reproduce

./rdkafka_example -P -t test -b 10.24.3.180:9092  -X security.protocol=sasl_ssl -X ssl.ca.location=../tests/ca-cert -X ssl.certificate.location=../tests/client_0client.pem -X ssl.key.location=../tests/client_0client.key  -X ssl.key.password=abcdefgh -X sasl.username=client -X sasl.password=client-secret -X sasl.mechanism=PLAIN -d all

From the previous log we can see that the metadata is not fetched correctly and keep trying to update it then produce data failed ,but I can't find the reason.

I have checked that use kafka console produer, I have write data successfully:

./bin/kafka-console-producer.sh --broker-list 10.23.4.180:9092  --topic test  --producer.config config/client_security.properties

The content of client_security.properties:

security.protocol=SASL_SSL
ssl.truststore.location=/work_space/open-source/librdkafka-1.2.1/tests/broker_0server.truststore.jks
ssl.truststore.password=abcdefgh
sasl.mechanism=PLAIN
ssl.endpoint.identification.algorithm=
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username=\"client\" \
    password=\"client-secret\";

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

theidexisted commented 4 years ago

After remove this configuration filed from server.properity: advertised.host.name=10.23.4.180, the problem is gone.