confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
21 stars 960 forks source link

MSSQL source connector does not query any rows #554

Open unfw opened 5 years ago

unfw commented 5 years ago

The Kafka connector do not seem to be querying any rows from a view that I set up and therefore not producing any data. My cluster allow for the auto creation of schemas but I have also tried to create the schema before starting the connector to see if that was the issue.

My configuration is:

{
  "name": "Payments-RichContract",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.password": "REDACED",
    "topic.prefix": "R2_",
    "value.converter": "io.confluent.connect.avro.AvroConverter",
    "value.converter.schema.registry.url": "http://localhost:8081",
    "key.converter": "io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url": "http://localhost:8081",
    "connection.url": "jdbc:sqlserver://REDACTED.database.windows.net:1433;database=wmp-contract;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30",
    "connection.user": "REDACTED",
    "table.whitelist": "Payments_Contract",
    "mode": "timestamp+incrementing",
    "incrementing.column.name": "ContractIID",
    "timestamp.column.name": "UpdatedTimeStamp",
    "table.types": "VIEW",
    "auto.register.schemas": "true"
  }
} 

I have validated that each of the column names and the table name is correct along with the connection details because I have used the same db connection details in several sink connectors that worked fine to get data into the database.

When looking at the connect logs, I only see it flushing 0 rows and there is no stack trace and it appears that the task is working without crashing.

Am I missing anything in this configuration?

rmoff commented 5 years ago

When you run the connector, what is the status of the tasks? You can use the REST API to query this, e.g.:

$ curl -s "http://localhost:8083/connectors"| jq '.[]'| \
    xargs -I{connector_name} curl -s "http://localhost:8083/connectors/"{connector_name}"/status"| \
    jq -c -M '[.name,.connector.state,.tasks[].state]|join(":|:")'| column -s : -t| sed 's/\"//g'| sort
connector_02  |  RUNNING  |  RUNNING