search

confluentinc / kafka-rest

Confluent REST Proxy for Kafka
https://docs.confluent.io/current/kafka-rest/docs/index.html
Other
38 stars 642 forks source link

JSON messages produced to protobuf topics fail after evolution #1260

Open noahseger opened 6 months ago

noahseger commented 6 months ago

This is actually a problem with the schema utils not allowing or default to ignoringUnknownFields: https://github.com/confluentinc/schema-registry/blob/20a98f3c4af1ffd804d8e41a2f274d48b1418b97/protobuf-provider/src/main/java/io/confluent/kafka/schemaregistry/protobuf/ProtobufSchemaUtils.java#L103-L108

What we expect: When producing protobufs as JSON with new (compatible) fields to Kafka, those fields are ignored before the latest registered schema is evolved to include them

What happens: Bad Request: Cannot find field: XXX in message

noahseger commented 6 months ago

Exactly where this problem occurs in REST proxy: https://github.com/confluentinc/kafka-rest/blob/master/kafka-rest/src/main/java/io/confluent/kafkarest/controllers/SchemaRecordSerializerImpl.java#L123