[question] Can I use schema-registry without Kafka broker independently?

[archenroot]

Hi guys,

I am building new microservices architecture. Inter connection of services itself is via MB (ActiveMQ in our case). BPM engine also use fully async patter via MB to communicate with service layers. For some clients I expose REST API. All is JSON driven.

In every microservice each request and generated response is going to be validated against schema as common step.

Is it valid to use schema-registry for my use as central tool from where schemas could be obtained and used for validation?

I am asking as I have no Kafka broker running at my sites in the moment, so is this product usable independently?

Thank you.

[ewencp]

@archenroot Great question. You can definitely use schema registry to manage the format of data outside of Kafka. That's why we use slightly different terminology in the schema registry ("subjects") than we use in Kafka ("topics" with separate formats for "key" and "value" in each message in the topic).

That said, out of the box we provide serializers for Kafka that help serialize the data by attaching a schema ID (in other systems you would have to implement this piece yourself), and currently the schema registry expects its storage to be backed by Kafka. So at the moment you'll still need a Kafka cluster to run the schema registry, but you can absolutely use it to manage the format of data in other data storage systems.

[archenroot]

@ewencp Thanks for quick and clear answer. Unfortunately I am not introducing Kafka in current architecture, we have quite "large objects (300k to 100M) & low count(100/1000s per day)" to process, so not optimal from my perspective for Kafka and as mentioned in the moment message broker will serve to transport events for middleware and logging for quite low number of systems, so we won't reach some default node cap at ~20,000 msg/s from performance perspective (and I will be running anyway in network of brokers cluster -> load balanced -> much higher throughput). On top of that I need to interface with external systems running on AMQP protocol, which when using Kafka over ActiveMQ will additionally require me to introduce AMQP-to-Kafka Bridge (running AMQP servers interfaces in front of Kafka clusters... :-)) )

Another option is building kafka cluster just for Schema Registry.. I am using ZooKeeper anyway for service discovery of some other services, so maybe will consider to enable only single 2 node Kafka cluster... will see.

Thanks again.

[ewencp]

@archenroot We don't support backing the schema registry with anything but Kafka today. It's possible it could be made more pluggable, and in fact the code already is somewhat pluggable via this interface: https://github.com/confluentinc/schema-registry/blob/master/core/src/main/java/io/confluent/kafka/schemaregistry/storage/KafkaStore.java However, Kafka (and an in-memory version for testing) are the only implementations available.

[tjwp]

@archenroot FYI we reimplemented the schema registry API as a Rails application using Postgres to store the schemas: https://github.com/salsify/avro-schema-registry.

[archenroot]

@tjwp well that is quite interesting as I have PostgreSQL anyway, but not introducing Kafka, so defacto working alternative for my case. I am not big fun of Rails :-) as I do every stuff related to frontend on nodejs and angular, but why not :-) hehe, I will give it a try.

Thanks a lot for great tip.

[ewencp]

I'm going to close this up as I think the original question has been answered. If someone wanted to look into it, as I said before it should be pretty easy to make the KafkaStore implementation pluggable.

[codebased]

While it is an old question, I am keen to implement my own schema registry API using C# and document Db. Is there any lead , which interface I should implement for the API in C#?