narcisoguillen / kafka-node-avro

ISC License
26 stars 13 forks source link

Evolve schema registry #19

Closed migh1 closed 4 years ago

migh1 commented 4 years ago

There's comment in the NPM page: Once schema was fetched from the registry it will keep it on memory to be re used.

I'm facing an issue when topic has a new schema evolution. The consumer can't get the new schema in runtime.

There's anyway to refetch/refresh the schemas?

I did the sames tests using this lib: https://www.npmjs.com/package/kafka-avro and worked fine evolving schemas an reading the new value in runtime.

narcisoguillen commented 4 years ago

Hey thanks for using kafka-node-avro, good point, might need to add or implement a way to clear or purge schemas, once init on the callback you have kafka namespace witch has the schemas as kafka.schemas , you can clean ( remove ) the schemas from there manually like :

KafkaAvro.init(Settings).then( kafka => {

  let schemaName = 'my.topic';

  kafka.send({
    topic    : schemaName,
    messages : { foo : 'bar' }
  }).then( sent => {

    // Eventually at some point, we want to remove a schema
    let schemaId = kafka.schemas[schemaName].id;

    delete kafka.schemas[schemaName]; // for producers
    delete kafka.schemas[schemaId]    // for consumers

  }, error => {
    console.error(error);
  });

} , error => {
  console.log(error);
});

kafka.schemas is the schema pool for all the lib, feel free to play around with it.

migh1 commented 4 years ago

Thank you for your answer, I'll try to do something related using kafka.schemas