I use the S3 Sink Connector on a topic with Avro records which are fully transitive compatible. I want to achieve that records are always written in larger batches to S3 in Parquet format. This works perfectly if the schema is identical. However, I was only able to keep the connector adding records to the same batch, if either the schema version of the new record is larger or equal (FORWARD), or if the schema version of the new record is less or equal (BACKWARD). I would expect if I set schema.compatibility to "FULL", that it keeps adding to the batch independent of the schema version. However, this is not the case. It treats "FULL" exactly the same as "BACKWARD": https://github.com/confluentinc/kafka-connect-storage-common/blob/092297f650552433afd2bfa0ba2280137aa861c5/core/src/main/java/io/confluent/connect/storage/schema/StorageSchemaCompatibility.java#L107
I would expect that "FULL" overrides "checkVersions" and always returns true.
Do you agree with this, or did I understood something wrong?
Hello,
I use the S3 Sink Connector on a topic with Avro records which are fully transitive compatible. I want to achieve that records are always written in larger batches to S3 in Parquet format. This works perfectly if the schema is identical. However, I was only able to keep the connector adding records to the same batch, if either the schema version of the new record is larger or equal (FORWARD), or if the schema version of the new record is less or equal (BACKWARD). I would expect if I set schema.compatibility to "FULL", that it keeps adding to the batch independent of the schema version. However, this is not the case. It treats "FULL" exactly the same as "BACKWARD": https://github.com/confluentinc/kafka-connect-storage-common/blob/092297f650552433afd2bfa0ba2280137aa861c5/core/src/main/java/io/confluent/connect/storage/schema/StorageSchemaCompatibility.java#L107
I would expect that "FULL" overrides "checkVersions" and always returns true. Do you agree with this, or did I understood something wrong?