ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector)

[ericfran]

Hi, we see this in the Schema-Registry logs, after some hours after be started the service seems to be "stopped"

Log enty: [2022-11-08 20:10:56,188] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-key, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource) [2022-11-08 20:10:56,191] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-key/versions HTTP/1.1" 200 12 6 (io.confluent.rest-utils.requests) [2022-11-08 20:10:56,192] INFO Registering new schema: subject srt_prd_01.srt.enterprise_store_deployments-value, version null, id null, type null (io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource) [2022-11-08 20:10:56,194] INFO 172.19.129.240 - - [08/Nov/2022:19:10:56 +0000] "POST /subjects/srt_prd_01.srt.enterprise_store_deployments-value/versions HTTP/1.1" 200 12 3 (io.confluent.rest-utils.requests) [2022-11-08 20:11:19,089] INFO Stopped NetworkTrafficServerConnector@3b94d659{HTTP/1.1, (http/1.1)}{hrxkfpdc01.hrx.erp:8081} (org.eclipse.jetty.server.AbstractConnector) [2022-11-08 20:11:19,090] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session) [2022-11-08 20:11:19,092] INFO Stopped o.e.j.s.ServletContextHandler@4ed5eb72{/ws,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler) [2022-11-08 20:11:19,100] INFO Stopped o.e.j.s.ServletContextHandler@12f9af83{/,null,STOPPED} (org.eclipse.jetty.server.handler.ContextHandler) [2022-11-08 20:11:19,103] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry) [2022-11-08 20:11:19,103] INFO [kafka-store-reader-thread-_schemas]: Shutting down (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)

[2022-11-08 20:11:19,108] INFO [kafka-store-reader-thread-_schemas]: Shutdown completed (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread) [2022-11-08 20:11:19,110] INFO KafkaStoreReaderThread shutdown complete. (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread) [2022-11-08 20:11:19,112] INFO Kafka store producer shut down (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2022-11-08 20:11:19,112] INFO Kafka store shut down complete (io.confluent.kafka.schemaregistry.storage.KafkaStore) [2022-11-08 20:11:19,114] ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector) org.apache.kafka.common.errors.WakeupException at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:514) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:278) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:227) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.awaitMetadataUpdate(ConsumerNetworkClient.java:164) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:257) at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.ensureCoordinatorReady(SchemaRegistryCoordinator.java:223) at io.confluent.kafka.schemaregistry.leaderelector.kafka.SchemaRegistryCoordinator.poll(SchemaRegistryCoordinator.java:108) at io.confluent.kafka.schemaregistry.leaderelector.kafka.KafkaGroupLeaderElector$1.run(KafkaGroupLeaderElector.java:202) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [2022-11-08 20:11:20,025] INFO SchemaRegistryConfig values: access.control.allow.headers = access.control.allow.methods = access.control.allow.origin = access.control.skip.options = true authentication.method = NONE authentication.realm = authentication.roles = [*] authentication.skip.paths = [] avro.compatibility.level = FULL compression.enable = true csrf.prevention.enable = false csrf.prevention.token.endpoint = /csrf csrf.prevention.token.expiration.minutes = 30 csrf.prevention.token.max.entries = 10000 debug = false host.name = hrxkfpdc01.hrx.erp idle.timeout.ms = 30000 inter.instance.headers.whitelist = [] inter.instance.protocol = http kafkastore.bootstrap.servers = [PLAINTEXT://hrxkfpdc01.hrx.erp:9092] kafkastore.checkpoint.dir = /tmp kafkastore.checkpoint.version = 0 kafkastore.connection.url = kafkastore.group.id = kafkastore.init.timeout.ms = 60000 kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit kafkastore.sasl.kerberos.min.time.before.relogin = 60000 kafkastore.sasl.kerberos.service.name = kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05 kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8 kafkastore.sasl.mechanism = GSSAPI kafkastore.security.protocol = PLAINTEXT kafkastore.ssl.cipher.suites = kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1 kafkastore.ssl.endpoint.identification.algorithm = kafkastore.ssl.key.password = [hidden] kafkastore.ssl.keymanager.algorithm = SunX509 kafkastore.ssl.keystore.location = kafkastore.ssl.keystore.password = [hidden] kafkastore.ssl.keystore.type = JKS kafkastore.ssl.protocol = TLS kafkastore.ssl.provider = kafkastore.ssl.trustmanager.algorithm = PKIX kafkastore.ssl.truststore.location = kafkastore.ssl.truststore.password = [hidden] kafkastore.ssl.truststore.type = JKS

What can be the root cause ?

Thanks

[OneCricketeer]

WakeupException is a general Consumer error, which would imply your Kafka broker is most likely the root problem.

Please describe your installation setup

[OneCricketeer]

Looking at host.name and kafkastore.bootstrap.servers in your output, you have the Registry running on the same server as one of your brokers, which is not recommended. Also, if that address is not fault tolerant, you should provide multiple broker addresses as your bootstrap servers.