We have two columns in our view id (type number) and date (type Date) which are both defined as non-nullable columns. On kafka they are registered as primitive int and long (as logicalType timestamp-millis) respectively, as expected.
The issue we are facing is that, from time to time, despite having no changes in the table/view definition the connector tries to register these two attributes as type UNION with null as default value. Since we have set the compatibility to FULL this causes the connector to fail everytime this happens. If we let it through by setting compatibility to NONE, we have noticed that later at some point it flips back to original types (as we expect and as its supposed to be). This back and forth flipping is causing a lot of issues for us.
We have tried to debug this and saw that for some reason the metadata fetched from the DB thinks that these fields are nullable. But like mentioned above there has been no DDL changes and the view definition also confirms that these two columns are non-nullable.
Is this a known issue or has anyone else encountered a similar problem and know what the issue could be and how to solve it?
We have a set up where we use JDBC source connector to get data from our Oracle database. The following is the connector configuration used:
{ "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max": "N", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "key.converter.schemas.enable": "false", "value.converter": "io.confluent.connect.avro.AvroConverter", "value.converter.schema.registry.url": "http://schema-registry.url....", "connection.url": "jdbc:oracle:thin:@ldap://....", "connection.user": "abc", "connection.password": "xyz", "mode": "incrementing", "incrementing.column.name": "histUid", "topic.prefix": "dev-live-", "numeric.mapping": "best_fit", "table.types": "VIEW", "poll.interval.ms": "10000", "validate.non.null": "false", "table.whitelist": "view1,view2,viewN", "table.poll.interval.ms": "3600000", "transforms": "createKey,extractKey,sbdTopicTransform", "transforms.createKey.type": "org.apache.kafka.connect.transforms.ValueToKey", "transforms.createKey.fields": "id", "transforms.extractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key", "transforms.extractKey.field": "id", "transforms.sbdTopicTransform.type": "com.swisscom.sbd.kafka.connect.transform.FirehoseTopicName" }
We have two columns in our view id (type number) and date (type Date) which are both defined as non-nullable columns. On kafka they are registered as primitive int and long (as logicalType timestamp-millis) respectively, as expected.
The issue we are facing is that, from time to time, despite having no changes in the table/view definition the connector tries to register these two attributes as type UNION with null as default value. Since we have set the compatibility to FULL this causes the connector to fail everytime this happens. If we let it through by setting compatibility to NONE, we have noticed that later at some point it flips back to original types (as we expect and as its supposed to be). This back and forth flipping is causing a lot of issues for us.
We have tried to debug this and saw that for some reason the metadata fetched from the DB thinks that these fields are nullable. But like mentioned above there has been no DDL changes and the view definition also confirms that these two columns are non-nullable.
Is this a known issue or has anyone else encountered a similar problem and know what the issue could be and how to solve it?