Better management and documentation of protobuf field numbers in Kafka Connect converter / source

[james-johnston-thumbtack]

Protobuf is unique among Schema Registry formats in that the fields are also numbered in addition to named. And the choice of number is extremely important from a protobuf compatibility standpoint.

However, how the Kafka Connect sources manage these protobuf field numbers is not at all clear to me. The documentation doesn't say much about the matter at all:

• The page discussing the protobuf Connect converter makes no mention of field numbers (searching the page for "number", "tag", and "ID").
• The page dedicated to the protobuf serdes makes no mention either (searching the page for "number", "tag" and "ID").
• The page discussing compatibility checks makes a general comment about field numbers being used to check compatibility, but nothing is said about how it pertains to Kafka Connect sources.

In reality, it looks like a Kafka Connect source would pick field numbers in an auto-incrementing fashion: https://github.com/confluentinc/schema-registry/blob/a12d763bf6813791065a3b0036f4a8eec28f71ed/protobuf-converter/src/main/java/io/confluent/connect/protobuf/ProtobufData.java#L729

This seems like it has a lot of consequences from a compatibility standpoint:

• Suppose our connector is reading from a PostgreSQL database, and a column is dropped in the original schema. Then all of the protobuf field numbers for the following columns would be renumbered by the converter, which isn't a compatible change.
• Similarly if a new column was inserted into the field list (rather than appended to the end), it would cause a renumbering of the fields.
• Thus the only way to safely read messages would be dynamically and not using code generated with protoc that might have fixed field numbers. But this is not always supported: https://github.com/confluentinc/confluent-kafka-go/issues/953 --- for example in this case, I do not understand how it is safe or even possible to read a protobuf-formatted topic originating from Kafka Connect in Golang.

How I would imagine this would work in source converters:

• The previous schema(s) would be used as a reference when generating a new schema in a Connect source. Field numbers would be reused rather than renumbering in incompatible ways.
• When a field is no longer used (has been deleted), it is marked as reserved, per best practices. (I notice that the word "reserve" does not appear anywhere in https://github.com/confluentinc/schema-registry/blob/a12d763bf6813791065a3b0036f4a8eec28f71ed/protobuf-converter/src/main/java/io/confluent/connect/protobuf/ProtobufData.java )
• Or in other words, when the Kafka Connect source numbers fields, it would be guaranteed never to reuse a field number that has been previously used in any previous schema. (Normally with protobuf this would be tracked with reserved keyword.)

But essentially, to start with, what I think is most missing is documentation around what the protobuf converter does when dynamically creating new schemas in a Kafka Connect source, especially around field numbers.

[rayokota]

@james-johnston-thumbtack , until we improve the Protobuf converter, possibly as you suggested, for now you can use an SMT to manually add the tags to the Connect schema. For each field in the Connect schema, you would add a parameter with key "io.confluent.connect.protobuf.Tag" and the value being the desired tag number.

[dmariassy]

Hi @rayokota , any progress on providing better support for the idiomatic use of protobuf field ids?