Schema registry support for kafka.pubsub component

[passuied]

Describe the feature

New to DAPR and really eager to leverage it at my company. I was surprised that there is no Schema registry support or Avro/protobuf serialization in the kafka.pubsub component. Is this on the roadmap? The way it could work is enabling configuring schema registry URL/Authentication. You could add metadata to the subscription to add specific overrides (e.g. schema name)

Release Note

RELEASE NOTE: ADD New feature in Dapr.

[passuied]

Seems there was a mention in https://github.com/dapr/components-contrib/issues/8#issuecomment-1191225512 but the issue was never created...

[berndverst]

It's not supported or on the roadmap, but we will accept contributions for this!

[passuied]

So updating this issue with findings. This work is not trivial as the current sarama Kafka Go implementation doesn't support Schema registry... Based on convo with @berndverst, we cannot switch to the Confluent Kafka client as it's leveraging a C library librdkafka and C bindings are not allowed in Dapr. So the solution might be to re-implement the kafka.pubsub component as a pluggable component, which doesn't have the same limitations... Doesn't seem trivial either but maybe easier?

[passuied]

Hi, so getting ready to help on this issue. I'm looking into adding Schema Registry to the kafka-pubsub component by using the https://github.com/riferrei/srclient library. I'm not experienced in Go but it seems to be pure Go and therefore compatible. Worst case scenario I could just re-implement the SchemaRegistryClient and leverage the underlying github.com/linkedin/goavro/v2 library for serialization/deserialization. I would focus on avro first, since this is our use case.

In terms of interface, I would:

• Add the following settings to the component:
  • schemaRegistryUrl: Schema Registry URL
  • schemaRegistryApiKey: Schema Registry credentials API Key
  • schemaRegitryApiSecret: Schema Registry credentials API Secret
  • autoRegisterSchemas: Whether the schema will get auto registered if missing upon publishing (default=True)
• Add the following metadata to the subscription metadata
  • valueSchemaType: Schema Type for value (None, AVRO only for now)

So a subscription would look something like this:

{'pubsubname': 'pubsub',
                      'topic': 'deathStarStatus',
                      'route': 'dsstatus',
                      'metadata': {
                          'valueSchemaType': 'AVRO'
                      } }

Assumptions:

• This will NOT be compatible with CloudEvents as while There is a Cloud Event Avro version, Schema Registry doesn't support dynamic types, so the data node will not be able to be passed -- apart from a string, negating any of the schema benefits...

Would love some feedback on the above @berndverst and others

[passuied]

@ItalyPaleAle @yaron2 should this issue receive the 1.13 label? I want to make sure it's included in upcoming version. I assume it will automatically be but just in case...

[ItalyPaleAle]

@ItalyPaleAle @yaron2 should this issue receive the 1.13 label? I want to make sure it's included in upcoming version. I assume it will automatically be but just in case...

Added. We do go through all PRs as part of the final hygiene however, and add labels as needed :)