search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.21k stars 1.11k forks source link

org.apache.kafka.common.errors.NotEnoughReplicasException #2641

Open Aliisshu19 opened 1 year ago

Aliisshu19 commented 1 year ago

Hi Team, Confluent Schema-registry pod in CrashLoopBackOff state because of below error: [2023-04-28 15:42:50,945+0000] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication) io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryInitializationException: Error initializing kafka store while initializing schema registry at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:314) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:75) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:90) at io.confluent.rest.Application.configureHandler(Application.java:284) at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:267) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44) Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreInitializationException: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store. at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:151) at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.init(KafkaSchemaRegistry.java:312) ... 6 more Caused by: io.confluent.kafka.schemaregistry.storage.exceptions.StoreException: Failed to write Noop record to kafka store. at io.confluent.kafka.schemaregistry.storage.KafkaStore.getLatestOffset(KafkaStore.java:493) at io.confluent.kafka.schemaregistry.storage.KafkaStore.waitUntilKafkaReaderReachesLastOffset(KafkaStore.java:296) at io.confluent.kafka.schemaregistry.storage.KafkaStore.init(KafkaStore.java:149) ... 7 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required. at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:97) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:79) at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:30) at io.confluent.kafka.schemaregistry.storage.KafkaStore.getLatestOffset(KafkaStore.java:488) ... 9 more Caused by: org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.

This issue is similar to https://github.com/confluentinc/schema-registry/issues/313. The workaround provided for this issue is to delete the "_schemas" topic and restart schema-registry pod. But after deleting the pod we are losing data. Is this issue identified and any proper fix coming in future release?

Aliisshu19 commented 1 year ago

Hi Team,

Any update on this issue?

Thanks

Aliisshu19 commented 1 year ago

Hi Team, Can I have any update on this issue?

knut-bw commented 3 months ago

I have the same issue, but I am deploying on Kubernetes. I believe the problem mainly occurs because the ReplicationFactor is less than min.insync.replicas. Besides deleting and recreating the topic, I have set the default ReplicationFactor to 3 since my min.insync.replicas is 2.

My Kafka YAML looks like this. This is useful to me.

apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
spec:
  configOverrides:
    server:
      - default.replication.factor=3
      - offsets.topic.replication.factor=3
  replicas: 3
  image:
    application: confluentinc/cp-server:7.6.0
    init: confluentinc/confluent-init-container:2.8.0
  dataVolumeCapacity: 10Gi
  podTemplate:
    envVars:
      - name: CLASSPATH
        value: /usr/share/java/kafka-connect-replicator/replicator-rest-extension-7.6.0.jar
    podSecurityContext:
      fsGroup: 1000
      runAsUser: 1000
      runAsNonRoot: true
  dependencies:
    zookeeper:
      endpoint: zookeeper.fcts.svc.cluster.local:2181

Hope this helps you.