search

Apicurio / apicurio-registry

An API/Schema registry - stores APIs and Schemas.
https://www.apicur.io/registry/
Apache License 2.0
609 stars 269 forks source link

Using Kafka for storage with mTLS security fails when trust store is PEM formatted #4975

Open simonra opened 3 months ago

simonra commented 3 months ago

Description

Registry Version "latest" (dockerhub image apicurio/apicurio-registry-kafkasql:latest-release
Persistence type "kafkasql"

For development and local testing I run a small Kafka cluster through docker compose, and I'd like to run Apicurio together with it. Given this I'd also like to use the Kafka cluster as storage for Apicurio because it's nice to have persistence across runs, but limits to how many storage technologies I'd like to have running in the background. The Kafka cluster is set up to use mTLS for security. However, when I try to get Apicurio to trust the Kafka clusters CA that I have in the form of a PEM certificate that I mount as the trust store, Apicurio fails during startup, likely due to the check in KafkaSqlFactory.tryToConfigureSecurity() at https://github.com/Apicurio/apicurio-registry/blob/main/app/src/main/java/io/apicurio/registry/storage/impl/kafkasql/KafkaSqlFactory.java#L328 . Because it requires both the trust store location, type, and password to be supplied to do anything, it either fails connecting to kafka due to the configs not being set, or fails during validation with the error Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: SSL trust store password cannot be specified for PEM format.

Note that the keystore probably suffers from the same problem (that is, if set up to be an unencrypted PEM because you'd store the password together with key in your secret management solution anyways, the code looks like that will fail in the same way.)

Environment

Starting it in docker compose.

What Version
Docker 27.0.3, build 7d4bcd8
Docker Compose v2.28.1-desktop.1
OS macOS Sonoma Version 14.5

Steps to Reproduce

  1. Have Kafka cluster with mTLS running in docker compose (e.g. https://github.com/NorskHelsenett/Kafka/blob/main/GetStarted/MultiBrokerClusterWithAuthAndMtls/docker-compose.yaml)
  2. Start Apicurio instance with config as show below
services:
  apicurio:
    image: apicurio/apicurio-registry-kafkasql:latest-release
    hostname: apicurio
    ports:
      - "8083:8083"
    volumes:
      - path/to/secrets/on-host:/kafka/secrets
    environment:
      REGISTRY_KAFKASQL_TOPIC_AUTO_CREATE: "true"
      REGISTRY_KAFKASQL_TOPIC: "apicurio-kafkasql-journal"
      KAFKA_BOOTSTRAP_SERVERS: "broker1:9092,broker2:9092,broker3:9092"

      APPLICATION_SERVER_PORT: "8083"
      APPLICATION_ID: "example-apicurioregistry"

      REGISTRY_KAFKASQL_SECURITY_PROTOCOL: "SSL"

      REGISTRY_KAFKASQL_SSL_KEYSTORE_LOCATION: "/kafka/secrets/acl-principal.pfx"
      REGISTRY_KAFKASQL_SSL_KEYSTORE_PASSWORD: "demo_cert_password"
      REGISTRY_KAFKASQL_SSL_KEYSTORE_TYPE: "PKCS12"
      # REGISTRY_KAFKASQL_SSL_TRUSTSTORE_PASSWORD: "null" # If this is not set the config doesn't load
      REGISTRY_KAFKASQL_SECURITY_SSL_TRUSTSTORE_TYPE: "PEM"
      REGISTRY_KAFKASQL_SECURITY_SSL_TRUSTSTORE_LOCATION: "/kafka/secrets/ca.crt"
  1. Startup fails

Expected vs Actual Behaviour

Actual behaviour: Apicurio fails to start when configured with PEM formatted trust store.

Expected behaviour: Apicurio starts and runs properly when configured with a PEM certificate file for trust store but no password.

Logs

Error logs when trust store password is set ``` INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/apicurio-registry-storage-kafkasql-2.6.1.Final-runner.jar INFO running in /deployments __ ____ __ _____ ___ __ ____ ______ --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \ --\___\_\____/_/ |_/_/|_/_/|_|\____/___/ 2024-08-05 08:42:34 INFO <> [null] (main) --------------------------------------------------------------------- 2024-08-05 08:42:34 INFO <> [null] (main) Legacy REST API date formats enabled (this is currently the default). 2024-08-05 08:42:34 INFO <> [null] (main) 2024-08-05 08:42:34 INFO <> [null] (main) For maximum compatibility and to ease upgrades from older versions 2024-08-05 08:42:34 INFO <> [null] (main) of Registry, the date format used in the REST API is not compliant 2024-08-05 08:42:34 INFO <> [null] (main) with OpenAPI standards (due to a bug in older versions). Please 2024-08-05 08:42:34 INFO <> [null] (main) make sure you upgrade all of your client applications to use the 2024-08-05 08:42:34 INFO <> [null] (main) latest client version. The next release will fix the date format 2024-08-05 08:42:34 INFO <> [null] (main) bug, which will result in older clients no longer being compatible 2024-08-05 08:42:34 INFO <> [null] (main) with the REST API. 2024-08-05 08:42:34 INFO <> [null] (main) 2024-08-05 08:42:34 INFO <> [null] (main) If you would like to fix the date format bug in THIS version of 2024-08-05 08:42:34 INFO <> [null] (main) Registry (great!) please set the following ENV variable + value: 2024-08-05 08:42:34 INFO <> [null] (main) 2024-08-05 08:42:34 INFO <> [null] (main) REGISTRY_APIS_V2_DATE_FORMAT=yyyy-MM-dd'T'HH:mm:ss'Z' 2024-08-05 08:42:34 INFO <> [null] (main) 2024-08-05 08:42:34 INFO <> [null] (main) Doing this will result in a REST API that is OpenAPI compliant, but 2024-08-05 08:42:34 INFO <> [null] (main) please remember to upgrade all your client applications first! 2024-08-05 08:42:34 INFO <> [null] (main) --------------------------------------------------------------------- 2024-08-05 08:42:34 WARN <> [null] (main) The 'quarkus.log.console.color' config property is deprecated and should not be used anymore 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) SqlRegistryStorage constructed successfully. JDBC URL: jdbc:h2:mem:registry_db 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is initialized. 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Database not initialized. 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Initializing the Apicurio Registry database. 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Database type: h2 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is up-to-date. 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Build's DB version is 15 2024-08-05 08:42:34 WARN <> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (main) Datasource '': JDBC resources leaked: 2 ResultSet(s) and 0 Statement(s) 2024-08-05 08:42:34 INFO <> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) Using Kafka-SQL artifactStore. 2024-08-05 08:42:34 INFO <> [org.apache.kafka.common.config.AbstractConfig] (main) AdminClientConfig values: auto.include.jmx.reporter = true bootstrap.servers = [broker1:9092, broker2:9092, broker3:9092] client.dns.lookup = use_all_dns_ips client.id = connections.max.idle.ms = 300000 default.api.timeout.ms = 60000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = GSSAPI sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SSL security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = /kafka/secrets/acl-principal.pfx ssl.keystore.password = [hidden] ssl.keystore.type = PKCS12 ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = /kafka/secrets/ca.crt ssl.truststore.password = [hidden] ssl.truststore.type = PEM 2024-08-05 08:42:34 INFO <> [org.apache.kafka.common.metrics.Metrics] (main) Metrics scheduler closed 2024-08-05 08:42:34 INFO <> [org.apache.kafka.common.metrics.Metrics] (main) Closing reporter org.apache.kafka.common.metrics.JmxReporter 2024-08-05 08:42:34 INFO <> [org.apache.kafka.common.metrics.Metrics] (main) Metrics reporters closed 2024-08-05 08:42:34 ERROR <> [io.quarkus.runtime.ApplicationLifecycleManager] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source) at io.quarkus.runtime.Application.start(Application.java:101) at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111) at io.quarkus.runtime.Quarkus.run(Quarkus.java:71) at io.quarkus.runtime.Quarkus.run(Quarkus.java:44) at io.quarkus.runtime.Quarkus.run(Quarkus.java:124) at io.quarkus.runner.GeneratedMain.main(Unknown Source) Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551) at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:488) at org.apache.kafka.clients.admin.Admin.create(Admin.java:134) at io.apicurio.registry.utils.kafka.KafkaUtil.createTopics(KafkaUtil.java:68) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage.autoCreateTopics(KafkaSqlRegistryStorage.java:200) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage.onConstruct(KafkaSqlRegistryStorage.java:136) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Bean.doCreate(Unknown Source) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Bean.create(Unknown Source) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35) at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c26(Unknown Source) at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35) at io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage_Observer_handleSqlStorageEvent_aD0SpxuOCT_mfMReL5VVVfvdQGw.notify(Unknown Source) at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346) at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:324) at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82) at io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage.initialize(AbstractSqlRegistryStorage.java:196) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at io.quarkus.arc.impl.Reflections.invokeMethod(Reflections.java:182) at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.doCreate(Unknown Source) at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.create(Unknown Source) at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35) at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.c56(Unknown Source) at io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances.computeIfAbsent(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35) at io.apicurio.registry.storage.impl.kafkasql.sql.KafkaSqlStore_Observer_onStart_GvKzwrQiqcFWu0a7W8wCqLRzan8.notify(Unknown Source) at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346) at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328) at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82) at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155) at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106) at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source) at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source) ... 7 more Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: SSL trust store password cannot be specified for PEM format. ```
Error logs when trust store password is not set ``` INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/apicurio-registry-storage-kafkasql-2.6.1.Final-runner.jar INFO running in /deployments __ ____ __ _____ ___ __ ____ ______ --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \ --\___\_\____/_/ |_/_/|_/_/|_|\____/___/ 2024-08-05 09:46:23 INFO <> [null] (main) --------------------------------------------------------------------- 2024-08-05 09:46:23 INFO <> [null] (main) Legacy REST API date formats enabled (this is currently the default). 2024-08-05 09:46:23 INFO <> [null] (main) 2024-08-05 09:46:23 INFO <> [null] (main) For maximum compatibility and to ease upgrades from older versions 2024-08-05 09:46:23 INFO <> [null] (main) of Registry, the date format used in the REST API is not compliant 2024-08-05 09:46:23 INFO <> [null] (main) with OpenAPI standards (due to a bug in older versions). Please 2024-08-05 09:46:23 INFO <> [null] (main) make sure you upgrade all of your client applications to use the 2024-08-05 09:46:23 INFO <> [null] (main) latest client version. The next release will fix the date format 2024-08-05 09:46:23 INFO <> [null] (main) bug, which will result in older clients no longer being compatible 2024-08-05 09:46:23 INFO <> [null] (main) with the REST API. 2024-08-05 09:46:23 INFO <> [null] (main) 2024-08-05 09:46:23 INFO <> [null] (main) If you would like to fix the date format bug in THIS version of 2024-08-05 09:46:23 INFO <> [null] (main) Registry (great!) please set the following ENV variable + value: 2024-08-05 09:46:23 INFO <> [null] (main) 2024-08-05 09:46:23 INFO <> [null] (main) REGISTRY_APIS_V2_DATE_FORMAT=yyyy-MM-dd'T'HH:mm:ss'Z' 2024-08-05 09:46:23 INFO <> [null] (main) 2024-08-05 09:46:23 INFO <> [null] (main) Doing this will result in a REST API that is OpenAPI compliant, but 2024-08-05 09:46:23 INFO <> [null] (main) please remember to upgrade all your client applications first! 2024-08-05 09:46:23 INFO <> [null] (main) --------------------------------------------------------------------- 2024-08-05 09:46:23 WARN <> [null] (main) The 'quarkus.log.console.color' config property is deprecated and should not be used anymore 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) SqlRegistryStorage constructed successfully. JDBC URL: jdbc:h2:mem:registry_db 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is initialized. 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Database not initialized. 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Initializing the Apicurio Registry database. 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Database type: h2 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Checking to see if the DB is up-to-date. 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.sql.AbstractSqlRegistryStorage] (main) Build's DB version is 15 2024-08-05 09:46:24 WARN <> [io.quarkus.agroal.runtime.AgroalEventLoggingListener] (main) Datasource '': JDBC resources leaked: 2 ResultSet(s) and 0 Statement(s) 2024-08-05 09:46:24 INFO <> [io.apicurio.registry.storage.impl.kafkasql.KafkaSqlRegistryStorage] (main) Using Kafka-SQL artifactStore. 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.config.AbstractConfig] (main) AdminClientConfig values: auto.include.jmx.reporter = true bootstrap.servers = [broker1:9092, broker2:9092, broker3:9092] client.dns.lookup = use_all_dns_ips client.id = connections.max.idle.ms = 300000 default.api.timeout.ms = 60000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 30000 retries = 2147483647 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.connect.timeout.ms = null sasl.login.read.timeout.ms = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.login.retry.backoff.max.ms = 10000 sasl.login.retry.backoff.ms = 100 sasl.mechanism = GSSAPI sasl.oauthbearer.clock.skew.seconds = 30 sasl.oauthbearer.expected.audience = null sasl.oauthbearer.expected.issuer = null sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000 sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000 sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100 sasl.oauthbearer.jwks.endpoint.url = null sasl.oauthbearer.scope.claim.name = scope sasl.oauthbearer.sub.claim.name = sub sasl.oauthbearer.token.endpoint.url = null security.protocol = SSL security.providers = null send.buffer.bytes = 131072 socket.connection.setup.timeout.max.ms = 30000 socket.connection.setup.timeout.ms = 10000 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.3] ssl.endpoint.identification.algorithm = https ssl.engine.factory.class = null ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.certificate.chain = null ssl.keystore.key = null ssl.keystore.location = /kafka/secrets/acl-principal.pfx ssl.keystore.password = [hidden] ssl.keystore.type = PKCS12 ssl.protocol = TLSv1.3 ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.certificates = null ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.utils.AppInfoParser$AppInfo] (main) Kafka version: 3.5.1 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.utils.AppInfoParser$AppInfo] (main) Kafka commitId: 2c6fb6c54472e90a 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.utils.AppInfoParser$AppInfo] (main) Kafka startTimeMs: 1722851184388 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.network.Selector] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Failed authentication with broker2/172.80.80.12 (channelId=-2) (SSL handshake failed) 2024-08-05 09:46:24 INFO <> [org.apache.kafka.clients.NetworkClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Node -2 disconnected. 2024-08-05 09:46:24 ERROR <> [org.apache.kafka.clients.NetworkClient] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Connection to node -2 (broker2/172.80.80.12:9092) failed authentication due to: SSL handshake failed 2024-08-05 09:46:24 WARN <> [org.apache.kafka.clients.admin.internals.AdminMetadataManager] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ... 19 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 24 more 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.utils.AppInfoParser] (kafka-admin-client-thread | adminclient-1) App info kafka.admin.client for adminclient-1 unregistered 2024-08-05 09:46:24 INFO <> [org.apache.kafka.clients.admin.internals.AdminMetadataManager] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Metadata update failed: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata 2024-08-05 09:46:24 INFO <> [org.apache.kafka.clients.admin.internals.AdminMetadataManager] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Metadata update failed: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata 2024-08-05 09:46:24 INFO <> [org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable] (kafka-admin-client-thread | adminclient-1) [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close. 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.metrics.Metrics] (kafka-admin-client-thread | adminclient-1) Metrics scheduler closed 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.metrics.Metrics] (kafka-admin-client-thread | adminclient-1) Closing reporter org.apache.kafka.common.metrics.JmxReporter 2024-08-05 09:46:24 INFO <> [org.apache.kafka.common.metrics.Metrics] (kafka-admin-client-thread | adminclient-1) Metrics reporters closed 2024-08-05 09:46:24 ERROR <> [io.quarkus.runtime.ApplicationLifecycleManager] (main) Failed to start application (with profile [prod]): java.lang.RuntimeException: Failed to start quarkus at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source) at io.quarkus.runtime.Application.start(Application.java:101) at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111) at io.quarkus.runtime.Quarkus.run(Quarkus.java:71) at io.quarkus.runtime.Quarkus.run(Quarkus.java:44) at io.quarkus.runtime.Quarkus.run(Quarkus.java:124) at io.quarkus.runner.GeneratedMain.main(Unknown Source) Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:571) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1413) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1344) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ... 19 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 24 more ```
apicurio-bot[bot] commented 3 months ago

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.