Inconsistent Schema Registry Behavior in Redpanda vs Confluent

[onursahin0]

Version & Environment

• Redpanda version (use rpk version): v24.1.9

• Docker

  redpanda-0:
  image: redpandadata/redpanda:v24.1.9
  command:
    - redpanda
    - start
    - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
    - --advertise-kafka-addr internal://redpanda-0:9092,external://127.0.0.1:19092
    - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
    - --advertise-pandaproxy-addr internal://redpanda-0:8082,external://127.0.0.1:18082
    - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
    - --rpc-addr redpanda-0:33145
    - --advertise-rpc-addr redpanda-0:33145
    - --mode dev-container
    - --smp 1
    - --overprovisioned
    - --default-log-level=info

Description

We have observed that Redpanda's Schema Registry behaves differently compared to Confluent's Schema Registry. Our .NET applications use Kafka Flow, specifically the ISchemaRegistryTypeNameResolver for resolving schema types. We integrate with Confluent's Schema Registry for our Kafka setup and expect Redpanda's Schema Registry to behave the same.

This schema registry resolver works perfectly with the Confluence Kafka in the production, however, when we run the same code against Redpanda, the code fails because the schema registry response is different than the Confluent Kafka.

Observations:

1. Schema Retrieval Format:
   • Confluent's Schema Registry retrieves schemas as base64-encoded proto descriptors.
   • Redpanda retrieves the schema string directly, necessitating additional conversion logic in our codebase to maintain compatibility.

For reference, here is a link to the relevant Kafka Flow code we are using for schema registry type name resolution: KafkaFlow ISchemaRegistryTypeNameResolver.

JIRA Link: CORE-5636

[github-actions[bot]]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.