search

FundingCircle / jackdaw

A Clojure library for the Apache Kafka distributed streaming platform.
https://fundingcircle.github.io/jackdaw/
BSD 3-Clause "New" or "Revised" License
369 stars 80 forks source link

Support for multiple schema on a topic #250

Open mt3593 opened 4 years ago

mt3593 commented 4 years ago

Hi,

We have a requirement to have different types of event on the same topic, this explains the need in great detail: https://www.confluent.io/blog/put-several-event-types-kafka-topic/

The functionality for doing this has been added to the confluent client via this PR https://github.com/confluentinc/schema-registry/pull/680

The current setup for using the avro serde is to provide a single schema. It would be quite nice to provide a list or possibly some configuration of these scheams for a topic.

I'm still working through what this will look like and what the test setup would be as you ideally want to preload all schemas into the mock-schema-registry and redef client to the mock on local test.

Is this something others would consider useful? Is this something jackdaw is planning to support?

mt3593 commented 4 years ago

So to get this to work on the current setup, you need to do something like:

(doto (.serializer (avro/serde schema-url schema false))
    (.configure {"schema.registry.url" schema-url
                        "value.subject.name.strategy" "SubjectNamingStrategy"}
                        false))

Where SubjectNamingStrategy is the fully qualified name of the class that implements getSubjectName as per the interface.

Ideally we should pass on config values into the serailizer rather than having to unpack the serde to set the config after.

cddr commented 4 years ago

Hey Mark,

Looks like you've been digging into this quite a lot :-) Thanks for your interest. I think there is an arity of the serde constructor which accepts a schema-registry config. It's used in this test helper here.

https://github.com/FundingCircle/jackdaw/blob/master/test/jackdaw/serdes/avro_test.clj#L33