confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client

https://github.com/confluentinc/confluent-kafka-dotnet/wiki

Apache License 2.0

88 stars 868 forks source link

.NET SchemaRegistry Protobuf support #1202

Open michaelpearce-gain opened 4 years ago

michaelpearce-gain commented 4 years ago

Description

Protobuf support has been added to schema registry (to be released still, but in master) https://github.com/confluentinc/schema-registry

It would be good to have protobuf serdes support in .net a once blocking issue that proto2 parts (descriptors) could not be accessed, but this was resolved in proto 3.11+.

Also it would be good to have dynamicmessage as like in Java

How to reproduce

Checklist

Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[ ] Confluent.Kafka nuget version.
[ ] Apache Kafka version.
[ ] Client configuration.
[ ] Operating system.
[ ] Provide logs (with "debug" : "..." as necessary in configuration).
[ ] Provide broker log excerpts.
[ ] Critical issue.

mhowlett commented 4 years ago

protobuf support is already essentially fully functional here: https://github.com/mhowlett/confluent-kafka-dotnet/tree/json_proto_sr2

it'll be in the release after 1.4 (which adds transactions)

we're not doing dynamicmessage to start with at least.

michaelandrepearce commented 4 years ago

@mhowlett thanks thats great news.

One more query, where we maybe sending protobuf in payload today. Is it possible that instead of framing (aka prepending the ids etc in the payload byte[]) possible to have a mode where payload is still just vanilla protobuf and the schema id etc are put in kafka headers.

Allowing an easier transition for those already just sending protobuf vanilla. E.g. i can start using the serializer today and not affect existing consumers and vice versa.

Same query prob applies to java variant also.

mhowlett commented 4 years ago

unfortunately not - this is dictated by schema registry.

there was a lengthy and very polarized internal discussion about this. the winning argument was essentially that having the metadata embedded is safer and easier to manage (no need to track 2 items everywhere).

michaelandrepearce commented 4 years ago

Would be good though if possible at least to toggle (e.g. keep default as framed, but have the option to use headers). Else makes migration where we have existing protobuf flows very hard.

mhowlett commented 4 years ago

fyi: @rayokota

michaelandrepearce commented 3 years ago

Is there any update on this? Having option to toggle putting schema id in header?