search

redpanda-data / console

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
https://redpanda.com
3.77k stars 346 forks source link

Configure RedPanda Console to use the AWS Avro Serializer #854

Open francescomucio opened 1 year ago

francescomucio commented 1 year ago

We are investigating the possibility to use Redpanda on top of our AWS MSK cluser where we publish Avro events. We learned the hard way that MSK doesn't ingest normal Avro events, but they needs to serialized with the custom AWS serializer (com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer).

Do we need to have a custom serializer if we use Redpanda with Avro events in MSK? And in case, is this a possibility?

weeco commented 1 year ago

We discussed this in the Redpanda Slack. Console does not support the AWS serializers yet and thus can not be configured to deserialize payloads that were serialized using this class yet.

francescomucio commented 1 year ago

Will it be possible to contribute to this repo adding this serializer?

weeco commented 1 year ago

Yes, ideally take a look at this: https://github.com/redpanda-data/console/tree/improve-message-viewer/backend/pkg/serde . This is a refactoring of the deserialization code which we plan to merge within the next few weeks. This should also make it easier to add a new deserialization strategy