confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.2k stars 1.11k forks source link

Schema registry failing to start due to mismatch in metadata #2707

Open sauravkpaul opened 1 year ago

sauravkpaul commented 1 year ago

we have 3 node cluster deployment with 3 kafka brokers and 2 schema-registry pods where the schema registry pods are failing to start after a upgrade and reboot. The error that we are getting is as below

{"log":"[2023-07-19 16:30:08,000] WARN [Producer clientId=producer-1] Received invalid metadata error in produce request on partition _schemas-0 due to org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)\n","stream":"stdout","time":"2023-07-19T16:30:08.000435821Z"}
{"log":"[2023-07-19 16:30:08,004] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)\n","stream":"stdout","time":"2023-07-19T16:30:08.010919567Z"}
{"log":"io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry\n","stream":"stdout","time":"2023-07-19T16:30:08.010966003Z"}

As per what I read, this issue can be mitigated if the producer in schema registry does multiple retries then the metadata will be corrected automatically corrected and initialization may happen. Is there a config I can try to increase the producer retries. Any other suggestion to recover from this issue. Thanks

maixiaohai commented 1 year ago

seems retry configuration was removed by this commit 63c59306d2dfbe349717cfe697e7f5831c74d553