Although I am able to open the truststore with the password mentioned in the mm2.properties using keytool on same pod. Getting below exception in Mirror Maker.
What could be the issue?
[2023-11-22 09:05:12,592] ERROR Scheduler for MirrorHeartbeatConnector caught exception in scheduled task: creating internal topics (org.apache.kafka.connect.mirror.Scheduler:102) org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:546) at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) at org.apache.kafka.connect.util.TopicAdmin.<init>(TopicAdmin.java:277) at org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:108) at org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:114) at org.apache.kafka.connect.mirror.MirrorHeartbeatConnector.createInternalTopics(MirrorHeartbeatConnector.java:83) at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93) at org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112) at org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /keystores/kafka-client-keystore.jks of type JKS at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161) at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:517) ... 14 more Caused by: java.io.IOException: keystore password was incorrect at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2090) at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243) at java.base/java.security.KeyStore.load(KeyStore.java:1479) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374) ... 24 more Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 28 more
Below is the content of mm2.properties from the pod.
Although I am able to open the truststore with the password mentioned in the mm2.properties using keytool on same pod. Getting below exception in Mirror Maker. What could be the issue?
[2023-11-22 09:05:12,592] ERROR Scheduler for MirrorHeartbeatConnector caught exception in scheduled task: creating internal topics (org.apache.kafka.connect.mirror.Scheduler:102) org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:546) at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) at org.apache.kafka.connect.util.TopicAdmin.<init>(TopicAdmin.java:277) at org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:108) at org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:114) at org.apache.kafka.connect.mirror.MirrorHeartbeatConnector.createInternalTopics(MirrorHeartbeatConnector.java:83) at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93) at org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112) at org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /keystores/kafka-client-keystore.jks of type JKS at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161) at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:517) ... 14 more Caused by: java.io.IOException: keystore password was incorrect at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2090) at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243) at java.base/java.security.KeyStore.load(KeyStore.java:1479) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374) ... 24 more Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 28 moreBelow is the content of mm2.properties from the pod.
Properties from environment
bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443 clusters=source,target emit.checkpoints.enabled=false emit.heartbeats.enabled=false groups= kafka.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443 kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks kafka.ssl.truststore.location=/keystores/truststore.jks port=tcp://10.97.51.255:7071 port.7071.tcp=tcp://10.97.51.255:7071 port.7071.tcp.addr=10.97.51.255 port.7071.tcp.port=7071 port.7071.tcp.proto=tcp replication.factor=3 security.protocol=SSL service.host=10.97.51.255 service.port=7071 service.port.metrics=7071 source->target.emit.heartbeats.enabled=false source->target.enabled=true source->target.sync.group.offsets=false source->target.topics=topic1 source.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443 source.config.storage.topic=namespace1.mm2.source.config.storage source.group.id=service1.namespace1.health-aggregator-service.mm.source source.kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks source.kafka.ssl.truststore.location=/keystores/truststore.jks source.offset.storage.topic=namespace1.mm2.source.offset.storage source.security.protocol=SSL source.status.storage.topic=namespace1.mm2.source.status.storage ssl.keystore.location=/keystores/kafka-client-keystore.jks ssl.truststore.location=/keystores/truststore.jks sync.topic.acls.enabled=false sync.topic.configs.enabled=false target->source.enabled=false target.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.iad-dataplane.prod.consul:443 target.config.storage.topic=namespace1.mm2.target.config.storage target.group.id=service1.namespace1.health-aggregator-service.mm.target target.kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks target.kafka.ssl.truststore.location=/keystores/truststore.jks target.offset.storage.topic=namespace1.mm2.target.offset.storage target.security.protocol=SSL target.status.storage.topic=namespace1.mm2.target.status.storage tasks.max=3 topic.replication.map=topic1=topic1
Connect workers (config topics)
ssl.truststore.password=HNIXZk7wsp ssl.keystore.password=kK9EEGs18B ssl.key.password=kK9EEGs18B
Connect sink connectors
consumer.ssl.truststore.password=HNIXZk7wsp consumer.ssl.keystore.password=kK9EEGs18B consumer.ssl.key.password=kK9EEGs18B
Connect source connectors
producer.ssl.truststore.password=HNIXZk7wsp producer.ssl.keystore.password=kK9EEGs18B producer.ssl.key.password=kK9EEGs18B