search

fd4s / fs2-kafka

Functional Kafka Streams for Scala
https://fd4s.github.io/fs2-kafka
Apache License 2.0
296 stars 100 forks source link

Reimplement schema registry client using http4s #506

Open bplommer opened 3 years ago

bplommer commented 3 years ago

For various reasons, the Java schema registry client has been causing classpath-related issues at work. It also has terrible diagnostics. The schema registry rest API is fairly straightforward so it's tempting to write our own implementation.

SystemFw commented 3 years ago

On top of that, the simple cache it uses is full of synchronized, so we've been thinking of reimplementing it as well

Wosin commented 3 years ago

Hey @bplommer, but the idea is to still imlement io.confluent.kafka.schemaregistry.client.SchemaRegistryClient inteface right ?

I be happy to tackle this btw.

bplommer commented 3 years ago

Hey @bplommer, but the idea is to still imlement io.confluent.kafka.schemaregistry.client.SchemaRegistryClient inteface right ?

That would be a good step in the right direction, yes.

Ideally I think I'd prefer to completely get rid of the Confluent libraries, implement a tagless final API for the schema registry, and implement fs2-kafka Serializer/Deserializer directly on top of that and the Avro serializer/deserializers, but that could be a further step.

I be happy to tackle this btw.

Thanks, that would be great!

Wosin commented 3 years ago

One more question, because I think I am missing something here :) We want to actually implement our own deserialization of Avro here? Since KafkaAvroDeserializer/Serializer are part of confluent libraries, so to completely get rid of that we would have to reimplement deserialization.

bplommer commented 3 years ago

One more question, because I think I am missing something here :) We want to actually implement our own deserialization of Avro here? Since KafkaAvroDeserializer/Serializer are part of confluent libraries, so to completely get rid of that we would have to reimplement deserialization.

That would be a much bigger task - it's not what I had in mind here (though I have a WIP PR for it at https://github.com/fd4s/vulcan/pull/289).

What I'm thinking is to use the regular Apache Avro serializers/deserializers that are used in e.g. vulcan.Codec.fromBinary and then we just re-implement the part of the binary protocol that's specific to schema registry - I guess we'd do that using either the ByteBuffer API or scodec-bits

keirlawson commented 3 years ago

Was taking a look at implementing this and was wondering, how feature-complete would such a client need to be to be worth merging? ie would you be looking for jsonschema/protobuf support alongside avro per the kafka one?

bplommer commented 2 years ago

Hi @keirlawson sorry for never replying to this! I think just having Avro support would be enough to be worth merging, though supporting json and protobuf too would also be good of course.

kevinmeredith commented 1 year ago

Hi @bplommer - the scope of this work is to provide a pure FP interface and implementation (using http4s) for both io.confluent.kafka.schemaregistry.client.SchemaRegistryClient and io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient?