search

lsst-sqre / strimzi-registry-operator

A Kubernetes Operator for running the Confluent Schema Registry with a Strimzi-based Kafka cluster
MIT License
81 stars 17 forks source link

Support for Schema CRDs #68

Open Purneau opened 2 years ago

Purneau commented 2 years ago

Hi all,

First of all, thumbs up for this great project.

I found your project because we are using a company-supplied Strimzi Kafka cluster. This company is on a cloud journey and still getting used to technologies like IaC, ArgoCD, etc. The new environment we set up is completely managed by ArgoCD. Because of this, we also want to manage our Topics and Schemas as code, but the latter seems to be a problem.

Do you see the need for this and/or do you have plans to support this in the near term? -- Full disclosure: I didn't test your setup yet, because I simply didn't see a Schema CRD in the Helm Chart, so I assume it does not exist --

Thanks a lot.

jonathansick commented 2 years ago

Hi @Purneau you're right that this project doesn't have a way of registering a schema via a Kubernetes CRD, like Strimzi's topic and user operators do for Kafka topics/users. This operator just manages the registry deployment and it's up to client apps to submit the schemas to the registry. That said, I find this idea of managing registry subjects/schemas via Kubernetes resources super interesting. I guess one would would have to think about how compatibility is managed and application development workflows, but it'd be an interesting thing to explore. I don't think this is in our roadmap at Rubin Observatory though, but maybe you or someone else could work on it? I think it'd be pretty orthogonal to what this operator does, so it could either be added here or via a brand new schema operator project.

What we do at Rubin is put our original sources of truth for schemas in our apps — usually the app that produces with a given registry subject. We use a "FORWARD" compatibility mode so that the producer can lead the way in upgrades. For Python, we've got a separate project called Kafkit that provides an API client for the schema registry. When our apps start up, they register their schemas with the registry and if the schema already exists it's a no-op but the app gets the schema ID. Managing schemas at the app level, rather than the deployment in Helm/Argo CD, seems to have worked well for us so far.