Not message in the topic

[cfuentesgithub]

Hello,

I have an issue, it seems that everything is correctly configured, but there are any message in the topic, i'm working with an Oracle 12c Database, but doesn't have multitenancy.

Any help?

This is my configuration name=oracle-logminer-connector connector.class=com.ecer.kafka.connect.oracle.OracleSourceConnector db.name.alias=gineta tasks.max=1 topic=testTopic db.name=gineta db.hostname=atacama71 db.port=1545 db.user=prueba db.user.password= db.fetch.size=1 table.whitelist=prueba.STREAM_TABLE_2,prueba.STREAM_TABLE parse.dml.data=true reset.offset=false start.scn=

this is part of the screen log. [2019-06-17 14:50:35,360] INFO StandaloneConfig values: access.control.allow.methods = access.control.allow.origin = bootstrap.servers = [logtrap.sii.cl:9092] header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter internal.key.converter = class org.apache.kafka.connect.json.JsonConverter internal.value.converter = class org.apache.kafka.connect.json.JsonConverter key.converter = class org.apache.kafka.connect.json.JsonConverter listeners = null metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 offset.flush.interval.ms = 5000 offset.flush.timeout.ms = 5000 offset.storage.file.filename = /tmp/connect.offsets plugin.path = null rest.advertised.host.name = null rest.advertised.listener = null rest.advertised.port = null rest.host.name = null rest.port = 8083 ssl.client.auth = none task.shutdown.graceful.timeout.ms = 5000 value.converter = class org.apache.kafka.connect.json.JsonConverter (org.apache.kafka.connect.runtime.standalone.StandaloneConfig:279)

[2019-06-17 14:50:36,463] INFO Kafka version : 1.1.1 (org.apache.kafka.common.utils.AppInfoParser:109) [2019-06-17 14:50:36,463] INFO Kafka commitId : 8e07427ffb493498 (org.apache.kafka.common.utils.AppInfoParser:110) [2019-06-17 14:50:36,469] INFO OracleSourceConnectorConfig values: db.fetch.size = 1 db.hostname = atacama71 db.name = gineta db.name.alias = gineta db.port = 1545 db.user = prueba db.user.password = pruebahdp parse.dml.data = true reset.offset = false start.scn = table.whitelist = prueba.STREAM_TABLE_2,prueba.STREAM_TABLE topic = dcv_topic (com.ecer.kafka.connect.oracle.OracleSourceConnectorConfig:279) [2019-06-17 14:50:36,470] INFO Oracle Kafka Connector is starting on gineta (com.ecer.kafka.connect.oracle.OracleSourceTask:82) [2019-06-17 14:50:36,470] INFO Connecting to database (com.ecer.kafka.connect.oracle.OracleSourceTask:84) [2019-06-17 14:50:36,470] INFO Created connector oracle-logminer-connector (org.apache.kafka.connect.cli.ConnectStandalone:104) [2019-06-17 14:50:36,986] INFO Starting LogMiner Session (com.ecer.kafka.connect.oracle.OracleSourceTask:89) [2019-06-17 14:50:37,188] INFO Getting current scn from database 4877431398519 (com.ecer.kafka.connect.oracle.OracleSourceTask:145) [2019-06-17 14:50:37,188] INFO Commit SCN : 0 (com.ecer.kafka.connect.oracle.OracleSourceTask:148) [2019-06-17 14:50:37,188] INFO Log Miner will start at new position SCN : 4877431398519 with fetch size : 1 (com.ecer.kafka.connect.oracle.OracleSourceTask:149) [2019-06-17 14:50:41,469] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:353) [2019-06-17 14:50:41,470] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:370) [2019-06-17 14:50:46,470] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:353) [2019-06-17 14:50:46,470] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:370) [2019-06-17 14:50:51,472] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:353) [2019-06-17 14:50:51,472] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:370) [2019-06-17 14:50:56,473] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:353) [2019-06-17 14:50:56,473] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:370)

[stariq2]

use reset.offset=true and you change it to false after you start looking at initial data.

[daanroosen-DS]

It seems that I am facing the same issue. It starts without errors but doesn't produce any messages. I already tried the proposed solution in the previous answer.

[2019-12-31 14:01:36,631] INFO Kafka version: 5.3.2-ccs (org.apache.kafka.common.utils.AppInfoParser) [2019-12-31 14:01:36,631] INFO Kafka commitId: 15a92ebe11136a76 (org.apache.kafka.common.utils.AppInfoParser) [2019-12-31 14:01:36,631] INFO Kafka startTimeMs: 1577800896631 (org.apache.kafka.common.utils.AppInfoParser) [2019-12-31 14:01:36,634] INFO [Worker clientId=connect-1, groupId=quickstart-avro] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [2019-12-31 14:01:36,644] INFO OracleSourceConnectorConfig values: db.fetch.size = 1 db.hostname = confluent db.name = XE db.name.alias = XE db.port = 1521 db.user = KMINER db.user.password = kminerpass multitenant = false parse.dml.data = true reset.offset = true start.scn = table.whitelist = KMINER.TEST topic = cdctest (com.ecer.kafka.connect.oracle.OracleSourceConnectorConfig) [2019-12-31 14:01:36,645] INFO Oracle Kafka Connector is starting on XE (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:36,676] INFO Connecting to database version 11.2.0.2.0 (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:36,676] INFO Starting LogMiner Session (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:36,747] INFO [Producer clientId=connector-producer-oracle-logminer-connector-0] Cluster ID: Tlj28QR9SKuyNZRfCd92cQ (org.apache.kafka.clients.Metadata) [2019-12-31 14:01:37,148] INFO Resetting offset with new SCN (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:37,151] INFO Getting current scn from database 379995 (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:37,151] INFO Commit SCN : 0 (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:01:37,151] INFO Log Miner will start at new position SCN : 379995 with fetch size : 1 (com.ecer.kafka.connect.oracle.OracleSourceTask) [2019-12-31 14:02:36,634] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-12-31 14:02:36,635] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-12-31 14:03:36,636] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-12-31 14:03:36,636] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-12-31 14:04:36,637] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)

Any suggestions?

Thanks by advance

[erdemcer]

Hi, It seems configuration is Ok. Same issue has occured for someone and he realized that no commit executed on DML. Could you please be sure about that ? If there is no issue about commit , does your consumer consume right topic ? Thanks.

[daanroosen-DS]

Thanks for your reply.

The commit was indeed the problem. The tool we were using wasn't doing autocommits..

[aaa55856]

Maybe I am facing the same issue. There are no message in the topic. I'm working with an Oracle 11g Database. And when I stop it with ctrl+c, it gives out some errors.

[2020-10-21 16:10:20,495] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser:117)
[2020-10-21 16:10:20,495] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser:118)
[2020-10-21 16:10:20,495] INFO Kafka startTimeMs: 1603267820495 (org.apache.kafka.common.utils.AppInfoParser:119)
[2020-10-21 16:10:20,505] INFO Created connector oracle-logminer-connector (org.apache.kafka.connect.cli.ConnectStandalone:112)
[2020-10-21 16:10:20,522] INFO OracleSourceConnectorConfig values:
        db.fetch.size = 1
        db.hostname = 192.168.0.120
        db.name = ORCL
        db.name.alias = ORCL.TEST
        db.port = 1521
        db.user = kminer
        db.user.password = kminerpass
        dml.types =
        multitenant = false
        parse.dml.data = true
        reset.offset = true
        start.scn =
        table.blacklist = TEST.*
        table.whitelist = ORCL.*
        topic = testdb
 (com.ecer.kafka.connect.oracle.OracleSourceConnectorConfig:347)
[2020-10-21 16:10:20,523] INFO Oracle Kafka Connector is starting on ORCL.TEST (com.ecer.kafka.connect.oracle.OracleSourceTask:113)
[2020-10-21 16:10:20,586] INFO [Producer clientId=connector-producer-oracle-logminer-connector-0] Cluster ID: Bg831uqoQn6XxI8C_WSh4g (org.apache.kafka.clients.Metadata:259)
[2020-10-21 16:10:20,955] INFO Connected to database version 112010 (com.ecer.kafka.connect.oracle.OracleSourceTask:118)
[2020-10-21 16:10:20,955] INFO Starting LogMiner Session (com.ecer.kafka.connect.oracle.OracleSourceTask:121)
[2020-10-21 16:10:20,970] INFO Resetting offset with new SCN (com.ecer.kafka.connect.oracle.OracleSourceTask:169)
[2020-10-21 16:10:20,972] INFO Getting current scn from database 2098575 (com.ecer.kafka.connect.oracle.OracleSourceTask:184)
[2020-10-21 16:10:20,972] INFO Commit SCN : 0 (com.ecer.kafka.connect.oracle.OracleSourceTask:187)
[2020-10-21 16:10:20,973] INFO Log Miner will start at new position SCN : 2098575 with fetch size : 1 (com.ecer.kafka.connect.oracle.OracleSourceTask:188)
[2020-10-21 16:10:30,505] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:416)
[2020-10-21 16:10:30,507] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:433)
[2020-10-21 16:10:40,508] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:416)
[2020-10-21 16:10:40,508] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:433)
[2020-10-21 16:10:50,509] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:416)
[2020-10-21 16:10:50,510] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:433)
^C[2020-10-21 16:10:53,161] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:66)
[2020-10-21 16:10:53,162] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:321)
[2020-10-21 16:10:53,168] INFO Stopped http_8083@666b83a4{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:343)
[2020-10-21 16:10:53,170] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:158)
[2020-10-21 16:10:53,173] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:338)
[2020-10-21 16:10:53,173] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:100)
[2020-10-21 16:10:53,174] INFO Stopping task oracle-logminer-connector-0 (org.apache.kafka.connect.runtime.Worker:704)
[2020-10-21 16:10:58,178] ERROR Graceful stop of task oracle-logminer-connector-0 failed. (org.apache.kafka.connect.runtime.Worker:736)
[2020-10-21 16:10:58,179] INFO Stopping connector oracle-logminer-connector (org.apache.kafka.connect.runtime.Worker:358)
[2020-10-21 16:10:58,186] INFO WorkerSourceTask{id=oracle-logminer-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:416)
[2020-10-21 16:10:58,187] INFO WorkerSourceTask{id=oracle-logminer-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:433)
[2020-10-21 16:10:58,190] ERROR WorkerSourceTask{id=oracle-logminer-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
org.apache.kafka.connect.errors.ConnectException: Error at database tier, Please check : java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

        at com.ecer.kafka.connect.oracle.OracleSourceTask.start(OracleSourceTask.java:219)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:208)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
[2020-10-21 16:10:58,192] ERROR WorkerSourceTask{id=oracle-logminer-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:180)
[2020-10-21 16:10:58,192] INFO Stop called for logminer (com.ecer.kafka.connect.oracle.OracleSourceTask:306)
[2020-10-21 16:10:58,192] INFO Logminer session cancel (com.ecer.kafka.connect.oracle.OracleSourceTask:309)
[2020-10-21 16:10:58,199] INFO Stopped connector oracle-logminer-connector (org.apache.kafka.connect.runtime.Worker:374)
[2020-10-21 16:10:58,199] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:198)
[2020-10-21 16:10:58,200] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:66)
[2020-10-21 16:10:58,201] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:219)
[2020-10-21 16:10:58,202] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:117)
[2020-10-21 16:10:58,202] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:71)

1. I copy kafka-connect-oracle-1.0.68.jar to $KAFKA_HOME/share/java/kafka
2. copy ojdbc6.jar to $KAFKA_HOME/libs(If I do not copy it into $KAFKA_HOME/libs instead of $KAFKA_HOME/share/java/kafka, it will fail to find the driver for database)
3. edited connect-standalone.properties : plugin.path=$KAFKA_HOME/share/java/kafka.(If I do not edit it, it will fail to find the kafka-connect-oracle class)
4. set $KAFKA_HOME/share/java/kafka in CLASSPATH There is no error report, but I also can not capture any changes.

If there any suggestions?

[erdemcer]

Hi, Connector is started successfully . Have you checked issues mentioned above which can be cause of your task ? You can ignore error messages when stopping connector . Thanks

[aaa55856]

Hi, Thanks for your reply!

Em... I think the database has commited the changes successfully, because I can read the changes in other management tool(Navicat) when I changes the database by shell(sqlplus).

I wonder if these WARNING matter?

[2020-10-23 16:57:33,895] INFO node0 Scavenging every 600000ms (org.eclipse.jetty.server.session:140)
Oct 23, 2020 4:57:34 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.
Oct 23, 2020 4:57:34 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource will be ignored.
Oct 23, 2020 4:57:34 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.RootResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.RootResource will be ignored.
Oct 23, 2020 4:57:34 PM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.LoggingResource will be ignored.
Oct 23, 2020 4:57:34 PM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method listLoggers in org.apache.kafka.connect.runtime.rest.resources.LoggingResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

[erdemcer]

Hi, Warning messages can be ignored . Is there any improvement regarding the issue ? Thanks

[CodeLulu318]

Hi, I also met the same problem. Have you solved it? Could you tell me the solution?

[jacksonmiller-devops]

Hi, I also met the same problem. Have you solved it? Could you tell me the solution? did you find any solution?