Unable to start schema-registry - Exception in schema registry group processing thread

[iceman91176]

Hi, we are not able to start our docker-based schema-registry. It starts up, but shuts down after about 30seconds with

[2019-11-12 20:46:35,052] INFO Started @10441ms (org.eclipse.jetty.server.Server)
[2019-11-12 20:46:35,052] INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain)
[2019-11-12 20:47:00,118] INFO Stopped NetworkTrafficServerConnector@2e570ded{HTTP/1.1,[http/1.1]}{0.0.0.0:8081} (org.eclipse.jetty.server.AbstractConnector)
[2019-11-12 20:47:00,118] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session)
[2019-11-12 20:47:00,120] INFO Stopped o.e.j.s.ServletContextHandler@a82c5f1{/ws,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2019-11-12 20:47:00,130] INFO Stopped o.e.j.s.ServletContextHandler@77102b91{/,null,UNAVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler)
[2019-11-12 20:47:00,131] INFO Shutting down schema registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry)
[2019-11-12 20:47:00,132] INFO [kafka-store-reader-thread-_schemas]: Shutting down (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2019-11-12 20:47:00,133] INFO [kafka-store-reader-thread-_schemas]: Shutdown completed (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2019-11-12 20:47:00,133] INFO [kafka-store-reader-thread-_schemas]: Stopped (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2019-11-12 20:47:00,134] INFO KafkaStoreReaderThread shutdown complete. (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread)
[2019-11-12 20:47:00,134] INFO [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2019-11-12 20:47:00,136] ERROR Unexpected exception in schema registry group processing thread (io.confluent.kafka.schemaregistry.masterelector.kafka.KafkaGroupMasterElector)
org.apache.kafka.common.errors.WakeupException
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeTriggerWakeup(ConsumerNetworkClient.java:498)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:284)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
        at io.confluent.kafka.schemaregistry.masterelector.kafka.SchemaRegistryCoordinator.poll(SchemaRegistryCoordinator.java:113)
        at io.confluent.kafka.schemaregistry.masterelector.kafka.KafkaGroupMasterElector$1.run(KafkaGroupMasterElector.java:192)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2019-11-12 20:47:00,138] WARN [Schema registry clientId=sr-1, groupId=my-schema-group] Close timed out with 1 pending requests to coordinator, terminating client connections (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)

We are running an ACL-enabled Kaflka-Cluster, master-coordination for schema-registry is handled in kafka. Thes issue seems to be related to #717, but we checked our acls. The kafka-principal that is used by the schema-registry can write,decribe, read the _schemas topic, it also has read access to the schema-registry group.

Whe looking at the timestamps it seems that the "Unexpected exception in schema registry group processing" is throws because schema-registry shuts down.

Any ideas ?

[iceman91176]

Sorry for raising that issue - you realized a stupid mistake that denied the kubernetes-healthcheck for schema-registry to work. So the container was restarted after 30 seconds ;-)

[hvitoi]

What was exactly the mistake? I'm facing exactly the same problem running cp-schema-registry:6.0.0 with ACL enabled on Kafka

[hvitoi]

Ok, nevermind, I just found the same mistake. It had to do with the livenessProbe and readinessProbe config in the Deployment file

[nilu-source]

how was this fixed?

[iceman91176]

@nilu-source it had to do with with the healtchecks in kubernetes. The healtcheck was blocked in our case and so kubernetes killed the container.....