search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.23k stars 1.11k forks source link

Schemas get deleted after compaction #1075

Open yogeshbelur opened 5 years ago

yogeshbelur commented 5 years ago

Hi

I have a scenario where system is running for long time, but after somedays i run console consumer on _schemas and i see 0 msgs. But when i query subjects from SR I get all the registered schema. I am afraid now that when i restart SR all schemas will vanish as currenlty SR has schemas in cache.

What is the reason schemas got deleted from topic.

Below are the topics settings

Topic:_schemas PartitionCount:1 ReplicationFactor:3 Configs:cleanup.policy=compact Topic: _schemas Partition: 0 Leader: 1001 Replicas: 1001,1002,1003 Isr: 1001,1003,1002

OneCricketeer commented 5 years ago

Are there still log files on the Kafka brokers for that topic?

jung-hyunchul commented 5 years ago

I have same issue. After restarting schema registry, I can't see any schema using schema-registry ui. so I consumed _schemas topic, but there were no schema at all. I have 3 zookeeper ( using docker ), 5 brokers ( using docker ), 1 schema registry ( using docker ) Kafka and schema registry are all same version 5.0.0

yogeshbelur commented 5 years ago

Hi sorry i had lost the logs. But i am trying to reproduce. I see _schemas logs and index delete message. Is it anything to do with log compact? Will not see messages on kafka topic when compact happened?

chaicesan commented 5 years ago

Any updates on this. We have experienced the same issue.

keithhawkridge commented 4 years ago

Any updates on this. We have experienced the same issue.

It seems that our schemas are deleted from the _schemas topic after a period of time

ArturKT commented 3 years ago

Any update? I have experienced this issue too.

ethanttbui commented 1 year ago

It baffles me that the is no response to such a serious issue..

renatocron commented 5 months ago

Any updates on this? I think I experienced a similar issue while running version confluentinc/cp-schema-registry:7.0.1.

I upgraded my cluster to larger machines, and everything was running fine for about 200 days, even after the upgrade. However, when I shut off the smaller pods, the schema registry pods relocated to the newer nodes. Somehow, the _schemas topic was the only one that appeared 'clean' after this.

This caused my producer to start creating new messages with new IDs, leading to a very cumbersome process to restore the messages back to their original IDs. I had to publish directly on _schemas while all producers, consumers, and the schema registry were scaled to 0, insert the duplicate magic numbers, restart the cluster, and then process the messages. I was fortunate to encounter only one collision where a previous magic ID matched a newer one, and this happened during a maintenance window.

I'm running 2 instances of the SR, without anything set besides SCHEMA_REGISTRY_HOST_NAME/SCHEMA_REGISTRY_LISTENERS/SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS so maybe some kind of race condition? maybe related to https://github.com/confluentinc/schema-registry/issues/3127 but I think that running in the same gruop is the default behavior, as I just one have 2 SR for HA