singhair
commented
6 years ago Can you paste the DDL that is working for you
Open rrangine opened 7 years ago
singhair
commented
6 years ago Can you paste the DDL that is working for you
I came across using the "timestamp" mode for a Oracle table's TIMESTAMP field with TIME ZONE in its DDL. Below is the case scenario :+1:
CREATE TABLE accounts(id INTEGER PRIMARY KEY NOT NULL, name VARCHAR(255), insert_date TIMESTAMP (6) WITH TIME ZONE DEFAULT ON NULL SYSTIMESTAMP NOT NULL ENABLE);
using the jdbc connect configuration: curl -XPOST -H 'Content-type:application/json' 'localhost:8083/connectors' -d '{ "name" : "jdbc-source", "config" : { "connector.class" : "io.confluent.connect.jdbc.JdbcSourceConnector", "tasks.max" : "1", "connection.url" : "jdbc:oracle:thin:user/secret@//host:port/service_name", "table.whitelist" : "ACCOUNTS",
"mode" : "timestamp", "timestamp.column.name" : " INSERT_DATE", "topic.prefix" : "oracle" } }'
Produces the below error the Connect cluster: [2017-08-22 17:26:35,352] INFO Source task WorkerSourceTask{id=jdbc-source-0} finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:143) [2017-08-22 17:26:35,355] WARN JDBC type -101 not currently supported (io.confluent.connect.jdbc.source.JdbcSourceTask:309) [2017-08-22 17:26:35,356] ERROR Task jdbc-source-0 threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:148) org.apache.kafka.connect.errors.DataException: INSERT_DATE is not a valid field name at org.apache.kafka.connect.data.Struct.lookupField(Struct.java:252) at org.apache.kafka.connect.data.Struct.get(Struct.java:74) at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractOffset(TimestampIncrementingTableQuerier.java:206) at io.confluent.connect.jdbc.source.TimestampIncrementingTableQuerier.extractRecord(TimestampIncrementingTableQuerier.java:182) at io.confluent.connect.jdbc.source.JdbcSourceTask.poll(JdbcSourceTask.java:208) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:163) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2017-08-22 17:26:35,356] ERROR Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:149) [2017-08-22 17:26:35,356] INFO Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer:972) [2017-08-22 17:26:36,034] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:825)
If, the DDL says with NO TIME ZONE the connect task interprets correctly. insert_date TIMESTAMP (6) WITH DEFAULT ON NULL SYSTIMESTAMP NOT NULL ENABLE
May be this needs to be fixed.
Raja Rangineni