search

confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
24 stars 959 forks source link

Problem using "quote.sql.identifiers" option when auto creating table #786

Open simonsteinmetz-adeo opened 4 years ago

simonsteinmetz-adeo commented 4 years ago

I'm using Confluent Platform version 5.4.0 and i'm facing the same problem than https://github.com/confluentinc/kafka-connect-jdbc/issues/455

With the option "quote.sql.identifiers":"always", it works but i need quote to request my table => select "myField" from "myTable"; If i change this option to "quote.sql.identifiers":"never", the connector raises this exception: org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception. at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:561) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:322) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at io.confluent.connect.jdbc.sink.DbStructure.amendIfNecessary(DbStructure.java:124) at io.confluent.connect.jdbc.sink.DbStructure.createOrAmendIfNecessary(DbStructure.java:74) at io.confluent.connect.jdbc.sink.BufferedRecords.add(BufferedRecords.java:121) at io.confluent.connect.jdbc.sink.JdbcDbWriter.write(JdbcDbWriter.java:66) at io.confluent.connect.jdbc.sink.JdbcSinkTask.put(JdbcSinkTask.java:74) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:539) ... 10 more

I found a workaround. If the destination table already exist and was correctly created (without quotes), then the connector works correctly only if using the option "quote.sql.identifiers":"never". (If you set this option to always, the connector could find the destination table).

This workaround doesn't work if the destination table doesn't exist and if using the option "auto.create": true to create the table. With these condition, the table is created using quotes....

Regards

singh-abhijeet commented 3 years ago

I am using JDBC Sink connector with "quote.sql.identifiers":"never" and "auto.create": true, and I see the table and attributes are created with quotes.

Can we use these configs together or they are designed to behave this way?