search

SAP / kafka-connect-sap

Kafka Connect SAP is a set of connectors, using the Apache Kafka Connect framework for reliably connecting Kafka with SAP systems
Apache License 2.0
125 stars 58 forks source link

Avro schema name in source connector is randomly recomputed with every query execution #145

Open pantaoran opened 1 year ago

pantaoran commented 1 year ago

When using this as a source connector with the queryMode set to query, then the name of the avro schema which gets registered in Schema Registry contains a random integer. This happens here in the code: https://github.com/SAP/kafka-connect-sap/blob/master/src/main/scala/com/sap/kafka/connect/source/querier/TableQuerier.scala#L95

This makes the connector unusable, because the name changes with every execution (apparently with every batch as specified by batch.max.rows, making it incompatible with previously registered versions of the schema. This then crashes the connector after the first batch was fetched, because it tries to set the same schema with a new name, which is incompatible with schema registry (unless using compatibility NONE, which I cannot in good conscience use in production).

This seems broken to me @elakito Can you please find a way to re-use the same schema name with every query? Maybe use a hash of the query or something?

pantaoran commented 1 year ago

More insights: this only happens when the queryMode is set to query. When I set it to table then it uses the table.name as the name for the avro schema, which is stable enough.