search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.21k stars 1.12k forks source link

_schemas topic replication factor gets changed during restart of kafka, zookeeper & schema-registry #956

Open abhijitdey56007 opened 5 years ago

abhijitdey56007 commented 5 years ago

Hi,

Our deployment model is a kubernetes based 3 node setup. Each node has one kafka broker and one zookeeper making up the entire cluster of 3 nodes. There is a exactly one schema-registry on one of the node s.

The following is the topic configuration in schema-registry. avro.compatibility.level=none kafkastore.topic=_schemas kafkastore.topic.replication.factor=3

The default.replication.factor configuration on kafka broker is set to 3.

After schema-registry is started, our deployment start-up script sets the cleanup.policy config to compact.

The replication factor for topic _schemas comes up as 3 as expected during start up of the deployment.

THe issue is that when all the nodes are restarted together (to simulate a power failure), the replication factor gets changed to 2 or 1 sometimes. We have many other topics but only for _schema it gets c hanged to 3.

Could you please help me out with this? Please let me know if you need any other information.

Regards,

Abhijit

hackmad commented 4 years ago

We have a similar kubernetes deployment and in non-production environment when there is failure, (a combination of kafka brokers, zookeepers, schema-registry restart), the _schemas topic suffers from a similar issue where replication factor changes to 1. However, min.insync.replicas stays at 2 and schema registry will fail to start.

Kubernetes/KOPS 1.15.6 Confluent Platform: 5.3.1-ccs (3 kafka brokers, 3 zookeeper nodes, 3 schema-registry - all statefulsets) Persistent Volumes: AWS EBS gp2 (300 IOPS)