search

averemee-si / oracdc

Oracle database CDC (Change Data Capture)
http://a2-solutions.eu/
Apache License 2.0
105 stars 36 forks source link

ORA-1291 when gap exist in V$ARCHIVED_LOG.SEQUENCE# #25

Closed rveve closed 2 years ago

rveve commented 2 years ago

Hi,

Data is only Loaded during Initial table data load to kafka topic after that Data is not going to the topic. But whenever I restarted the kafka connect cluster, a2 connector publishes the remaining data to the topic.

This is very similar to this issue: https://github.com/averemee-si/oracdc/issues/13

Thank you.

averemee-si commented 2 years ago

Looking into it...

Is it possible to upload full connector log?

Thanks, Aleksei

rveve commented 2 years ago 

(org.apache.kafka.connect.runtime.ConnectorConfig$EnrichedConnectorConfig:347)
[2022-01-14 19:37:30,663] INFO [Worker clientId=connect-1, groupId=cluster4-kafka-connect-cluster-oracle3-us-east-1-shared.aws] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1171)
[2022-01-14 19:37:30,719] INFO [Worker clientId=connect-1, groupId=cluster4-kafka-connect-cluster-oracle3-us-east-1-shared.aws] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1486)
[2022-01-14 19:37:31,133] WARN [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Persistent state file /tmp/oracdc.state not found! (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:337)
[2022-01-14 19:37:31,133] WARN [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] oracdc will use offset from Kafka cluster: SCN=6762383948205, RS_ID(RBA)=' 0x016408.000010bc.01c8 ', SSN=3 (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:338)
[2022-01-14 19:37:31,488] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Initializing oracdc initial load thread (eu.solutions.a2.cdc.oracle.OraCdcInitialLoadThread:61)
[2022-01-14 19:37:31,488] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] DB cores available 8, Kafka Cores available 1. (eu.solutions.a2.cdc.oracle.OraCdcInitialLoadThread:70)
[2022-01-14 19:37:31,488] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] {} parallel loaders for select phase will be used. (eu.solutions.a2.cdc.oracle.OraCdcInitialLoadThread:71)
[2022-01-14 19:37:31,492] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Initializing oracdc logminer archivelog worker thread (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:120)
[2022-01-14 19:37:31,739] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] LogMiner will start from SCN 6762383948205 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:93)
[2022-01-14 19:37:31,740] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Mining database BREWIQFRMN_BOSQ3G01DB201 is in OPEN mode (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:208)
[2022-01-14 19:37:31,740] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Same database will be used for dictionary query and mining (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:210)
[2022-01-14 19:37:31,740] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] RowPrefetch size for accessing V$LOGMNR_CONTENTS set to 32. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:228)
[2022-01-14 19:37:32,057] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91144_1014585229.arc thread# 1 sequence# 91144 first change number 6762383790977 next log first change 6762384043403 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174)
[2022-01-14 19:37:32,099] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Rewinding LogMiner ResultSet to first position after SCN = 6762383948205, RS_ID = ' 0x016408.000010bc.01c8 ', SSN = 3. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:253)
[2022-01-14 19:37:32,158] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Total records scipped while rewinding: 4, elapsed time ms: 1 (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:287)
[2022-01-14 19:37:32,158] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] BEGIN: OraCdcInitialLoadThread.run() (eu.solutions.a2.cdc.oracle.OraCdcInitialLoadThread:81)
[2022-01-14 19:37:32,160] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:220)
[2022-01-14 19:37:32,160] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] BEGIN: OraCdcLogMinerWorkerThread.run() (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:296)
[2022-01-14 19:37:32,220] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Chronicle core loaded from file:/usr/share/confluent-hub-components/a2solutions-oracdc-kafka/lib/chronicle-core-2.21.92.jar (net.openhft.chronicle.core.Jvm:178)
[2022-01-14 19:37:32,272] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Took 6 ms to add mapping for /tmp/BREWFR.EXAMPLE_TEST_ORA_CDC.5333593504814965936/metadata.cq4t (net.openhft.chronicle.bytes.MappedFile:56)
[2022-01-14 19:37:32,339] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Running under OpenJDK Runtime Environment 1.8.0_222-b10 with 1 processors reported. (net.openhft.chronicle.core.internal.announcer.InternalAnnouncer:56)
[2022-01-14 19:37:32,340] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Process id: 20 :: Chronicle Queue (5.21.93) (net.openhft.chronicle.core.internal.announcer.InternalAnnouncer:56)
[2022-01-14 19:37:32,342] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Analytics: Chronicle Queue reports usage statistics. Learn more or turn off: https://github.com/OpenHFT/Chronicle-Queue/blob/master/DISCLAIMER.adoc (net.openhft.chronicle.core.internal.announcer.InternalAnnouncer:56)
Client id file not present: /root/.chronicle.analytics.client.id java.nio.file.NoSuchFileException: /root/.chronicle.analytics.client.id
[2022-01-14 19:37:32,369] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Table BREWFR.EXAMPLE_TEST_ORA_CDC (DEPENDENCY='DISABLED') initial load (read phase) started. (eu.solutions.a2.cdc.oracle.OraTable4InitialLoad:558)
[2022-01-14 19:37:32,401] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Took 3.015 ms to pollDiskSpace for /tmp/BREWFR.EXAMPLE_TEST_ORA_CDC.5333593504814965936 (net.openhft.chronicle.threads.DiskSpaceMonitor:56)
[2022-01-14 19:37:32,968] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91145_1014585229.arc thread# 1 sequence# 91145 first change number 6762384043403 next log first change 6762384217383 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174)
[2022-01-14 19:37:33,798] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91146_1014585229.arc thread# 1 sequence# 91146 first change number 6762384217383 next log first change 6762384505214 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174)
[2022-01-14 19:37:34,370] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Table BREWFR.EXAMPLE_TEST_ORA_CDC initial load (read phase) completed. 1440 rows read. (eu.solutions.a2.cdc.oracle.OraTable4InitialLoad:573)
[2022-01-14 19:37:34,371] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] END: OraCdcInitialLoadThread.run(), elapsed time 2211 ms (eu.solutions.a2.cdc.oracle.OraCdcInitialLoadThread:112)
[2022-01-14 19:37:34,985] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91147_1014585229.arc thread# 1 sequence# 91147 first change number 6762384505214 next log first change 6762384804398 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174)
[2022-01-14 19:37:35,160] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Table BREWFR.EXAMPLE_TEST_ORA_CDC initial load (send to Kafka phase) started. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:578)
[2022-01-14 19:37:35,820] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Table BREWFR.EXAMPLE_TEST_ORA_CDC initial load (send to Kafka phase) completed. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:592)
[2022-01-14 19:37:36,899] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] Initial load completed (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:565)
[2022-01-14 19:37:44,061] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:37:44,061] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:37:44,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Finished commitOffsets successfully in 57 ms (org.apache.kafka.connect.runtime.WorkerSourceTask:529)
[2022-01-14 19:37:59,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:37:59,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:38:14,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:38:14,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:38:29,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:38:29,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:38:44,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:38:44,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.```
rveve commented 2 years ago

after that any change of the table, the connector log shows only

2022-01-14 19:38:14,118] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:38:29,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:38:29,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)
[2022-01-14 19:38:44,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)
[2022-01-14 19:38:44,119] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-006-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka. ```

Nothing goes to the topic until the connect cluster is restarted and then it publishes the extra data to the topic during the initial load.

Thank you for your quick response.

averemee-si commented 2 years ago

Thanks!

Could you please load connector properties file

Thanks, Aleksei

rveve commented 2 years ago

Basically, that is all we have here


{
  "connector.class": "eu.solutions.a2.cdc.oracle.OraCdcLogMinerConnector",
  "tasks.max": "1",
  "a2.process.lobs": "false",
  "a2.oracdc.schemas": "true",
  "a2.initial.load": "EXECUTE",
  "a2.topic.name.style": "SCHEMA_TABLE",
  "a2.jdbc.password": "REDACTED",
  "a2.include": "REDACTED",
  "confluent.topic.sasl.mechanism": "PLAIN",
  "a2.topic.name.delimiter": ".",
  "confluent.topic.bootstrap.servers": "bootstrap.servers",
  "a2.jdbc.url": "jdbc:oracle:thin:@(description=(address=(protocol=tcp)(host=REDACTED)(port=1621))(connect_data=(server=dedicated)(service_name=REDACTED)))",
  "a2.jdbc.username": "username",
  "value.converter.schema.registry.url": "https://schema.confluent.cloud",
  "a2.schema.type": "kafka"
}
rveve commented 2 years ago

It is a very simple config.

averemee-si commented 2 years ago

OK, thanks!

LogMiner connector worked with archived logs. I need to check contents of V$ARCHIVED_LOG... I will back soon with some easy queries against dynamic Oracle views.

Regards, Alexei

rveve commented 2 years ago

Sure, Thank you.

averemee-si commented 2 years ago

From log [2022-01-14 19:37:31,739] INFO [cluster4-BREWiqfrmn-a2-oracdc-006-nonprod|task-0] LogMiner will start from SCN 6762383948205 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:93) could you please provide output of select SEQUENCE#, FIRST_CHANGE#, NEXT_CHANGE#, NAME from V$ARCHIVED_LOG$ where FIRST_CHANGE# >= 6762383948205 and DELETED='NO' and ARCHIVED='YES'

Regards, Aleksei

FarooqWW commented 2 years ago

Please find output:

SQL> select SEQUENCE#, FIRST_CHANGE#, NEXT_CHANGE#, NAME from V$ARCHIVED_LOG where FIRST_CHANGE# >= 6762383948205 and DELETED='NO' and ARCHIVED='YES'

SEQUENCE# FIRST_CHANGE# NEXT_CHANGE# NAME

 91145    6762384043403   6762384217383 /archive/TESTDB/TESTDB_1_91145_1014585229.arc
 91146    6762384217383   6762384505214 /archive/TESTDB/TESTDB_1_91146_1014585229.arc
 91147    6762384505214   6762384804398 /archive/TESTDB/TESTDB_1_91147_1014585229.arc
 91148    6762384804398   6762385052755 /archive/TESTDB/TESTDB_1_91148_1014585229.arc
 91149    6762385052755   6762385295405 /archive/TESTDB/TESTDB_1_91149_1014585229.arc
 91150    6762385295405   6762385718491 /archive/TESTDB/TESTDB_1_91150_1014585229.arc
 91151    6762385718491   6762385952443 /archive/TESTDB/TESTDB_1_91151_1014585229.arc
 91152    6762385952443   6762386118890 /archive/TESTDB/TESTDB_1_91152_1014585229.arc
 91153    6762386118890   6762386235975 /archive/TESTDB/TESTDB_1_91153_1014585229.arc
 91154    6762386235975   6762386488851 /archive/TESTDB/TESTDB_1_91154_1014585229.arc
 91155    6762386488851   6762386590037 /archive/TESTDB/TESTDB_1_91155_1014585229.arc
 91156    6762386590037   6762386665359 /archive/TESTDB/TESTDB_1_91156_1014585229.arc
 91157    6762386665359   6762387113518 /archive/TESTDB/TESTDB_1_91157_1014585229.arc
 91158    6762387113518   6762387364488 /archive/TESTDB/TESTDB_1_91158_1014585229.arc
 91159    6762387364488   6762387879485 /archive/TESTDB/TESTDB_1_91159_1014585229.arc
 91160    6762387879485   6762387994177 /archive/TESTDB/TESTDB_1_91160_1014585229.arc
 91161    6762387994177   6762388265008 /archive/TESTDB/TESTDB_1_91161_1014585229.arc
 91162    6762388265008   6762388345064 /archive/TESTDB/TESTDB_1_91162_1014585229.arc
 91163    6762388345064   6762388647063 /archive/TESTDB/TESTDB_1_91163_1014585229.arc
 91164    6762388647063   6762388712784 /archive/TESTDB/TESTDB_1_91164_1014585229.arc
 91165    6762388712784   6762389291879 /archive/TESTDB/TESTDB_1_91165_1014585229.arc

21 rows selected.

averemee-si commented 2 years ago

In connector log: /archive/BREWIQFRMN/BREWIQFRMN_1_91145_1014585229.arc but in requested output /archive/TESTDB/TESTDB_1_91145_1014585229.arc

Are you sure you sent the query results from the same database?

FarooqWW commented 2 years ago

Yes I had just replaced the DBNAME during upload. its same query result. you can see from the sequence number.

averemee-si commented 2 years ago

Ok, thanks! Could you please check for presence of BREWFR.EXAMPLE_TEST_ORA_CDC changes in archived redo using steps below:

  1. Start LogMiner 1.1 If you are on 19.10+ please execute 
    begin
DBMS_LOGMNR.START_LOGMNR(
STARTSCN => 6762384043403,
 ENDSCN => 6762389291879,
 OPTIONS =>  
  DBMS_LOGMNR.SKIP_CORRUPTION +
  DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG +
  DBMS_LOGMNR.NO_SQL_DELIMITER +
  DBMS_LOGMNR.NO_ROWID_IN_STMT);
end;
/

    1.2 For RDBMS version < 19.10 please execute

    
begin
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91145_1014585229.arc' , OPTIONS => DBMS_LOGMNR.NEW);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91146_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91147_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91148_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91149_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91150_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91151_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91152_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91153_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91154_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91155_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91156_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91157_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91158_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91159_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91160_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91161_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91162_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91163_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91164_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91165_1014585229.arc' , OPTIONS => DBMS_LOGMNR.ADDFILE);
DBMS_LOGMNR.START_LOGMNR(
STARTSCN => 6762384043403,
 ENDSCN => 6762389291879,
 OPTIONS =>  
  DBMS_LOGMNR.SKIP_CORRUPTION +
  DBMS_LOGMNR.DICT_FROM_ONLINE_CATALOG +
  DBMS_LOGMNR.NO_SQL_DELIMITER +
  DBMS_LOGMNR.NO_ROWID_IN_STMT);

end; /

2. Check INSERT/UPDATE/DELETE/DDL for table BREWFR.EXAMPLE_TEST_ORA_CDC using

select OPERATION_CODE,XID,SQL_REDO from V$LOGMMNR_CONTENTS where OPERATION_CODE in (1,2,3,5) and SEG_OWNER='BREWFR' and TABLE_NAME='EXAMPLE_TEST_ORA_CDC' and rownum < 10;



Regards,
Aleksei
FarooqWW commented 2 years ago

the command is showing no data .

SYS@ BREWIQFRMN > select OPERATION_CODE,XID,SQL_REDO from V$LOGMNR_CONTENTS where OPERATION_CODE in (1,2,3,5) and SEG_OWNER='BREWFR' and TABLE_NAME='EXAMPLE_TEST_ORA_CDC' and rownum < 10;

no rows selected

SYS@ BREWIQFRMN >

However I have noticed that as soon as I am committing the records and switching logfile, data is being inserting into kafka topic now.

Let me re-deploy the connector from scratch and test this again. This was not working before for sure.

averemee-si commented 2 years ago

Hi Farooq,

Could you please also check

select SUPPLEMENTAL_LOG_DATA_MIN, SUPPLEMENTAL_LOG_DATA_PK, SUPPLEMENTAL_LOG_DATA_UI, SUPPLEMENTAL_LOG_DATA_FK, SUPPLEMENTAL_LOG_DATA_ALL
from V$DATABASE;

and

select LOG_GROUP_NAME, TABLE_NAME, DECODE(ALWAYS, 'ALWAYS', 'Unconditional', NULL, 'Conditional') ALWAYS
from DBA_LOG_GROUPS where TABLE_NAME='EXAMPLE_TEST_ORA_CDC';

Regards, Aleksei

FarooqWW commented 2 years ago 
select SUPPLEMENTAL_LOG_DATA_MIN, SUPPLEMENTAL_LOG_DATA_PK, SUPPLEMENTAL_LOG_DATA_UI, SUPPLEMENTAL_LOG_DATA_FK, SUPPLEMENTAL_LOG_DATA_ALL
from V$DATABASE;

SUPPLEMENTAL_LOG_DATA_MI SUPPLEMEN SUPPLEMEN SUPPLEMEN SUPPLEMEN
------------------------ --------- --------- --------- ---------
IMPLICIT         NO    NO        NO        YES

select LOG_GROUP_NAME, TABLE_NAME, DECODE(ALWAYS, 'ALWAYS', 'Unconditional', NULL, 'Conditional') ALWAYS from DBA_LOG_GROUPS where TABLE_NAME='EXAMPLE_TEST_ORA_CDC';

LOG_GROUP_NAME       TABLE_NAME             ALWAYS
-------------------- ------------------------------ ---------------------------------------
SYS_C0060465         EXAMPLE_TEST_ORA_CDC       Unconditional
averemee-si commented 2 years ago

Thanks.

Please make sure:

  1. You have set at database level 
    alter database add supplemental log data;
  2. You have set for table BREWFR .EXAMPLE_TEST_ORA_CDC 
    alter table BREWFR .EXAMPLE_TEST_ORA_CDC add supplemental log data (ALL) columns;

Hope, this helps. Don't hesitate to contact me in case of issues.

Regards, Aleksei

FarooqWW commented 2 years ago

Hi, I have verified above settings and they are already there. I am facing a new set of errors while deploying the connector. With Overnight backup, the old archives got purged from the the db server and we have just one day old archive logs. When I am trying to start connector, it starts adding few archivelogs and after that it throws following errors.

[2022-01-16 14:10:55,411] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2

The connector shows RUNNING Status but not loading anything in topics.

{ "name": "BREWiqfrmn-a2-oracdc-011-nonprod", "connector": { "state": "RUNNING", "worker_id": "0.0.0.0:8083" }, "tasks": [ { "id": 0, "state": "RUNNING", "worker_id": "0.0.0.0:8083" } ], "type": "source" }

I even tried to put start SCN in the connector config to start from latest timestamp however it is still throwing same errors.

"a2.first.change": "6762323596064",

Below is the full log for reference.

[2022-01-16 14:10:33,967] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91085_1014585229.arc thread# 1 sequence# 91085 first change number 6762359085412 next log first change 6762359291058 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:35,155] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91086_1014585229.arc thread# 1 sequence# 91086 first change number 6762359291058 next log first change 6762359441621 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:36,222] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91087_1014585229.arc thread# 1 sequence# 91087 first change number 6762359441621 next log first change 6762359558161 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:37,928] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91088_1014585229.arc thread# 1 sequence# 91088 first change number 6762359558161 next log first change 6762359788469 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:39,056] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91089_1014585229.arc thread# 1 sequence# 91089 first change number 6762359788469 next log first change 6762360475371 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:40,187] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91090_1014585229.arc thread# 1 sequence# 91090 first change number 6762360475371 next log first change 6762361128096 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:41,132] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91091_1014585229.arc thread# 1 sequence# 91091 first change number 6762361128096 next log first change 6762362712210 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:42,456] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91092_1014585229.arc thread# 1 sequence# 91092 first change number 6762362712210 next log first change 6762363293401 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:44,187] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91093_1014585229.arc thread# 1 sequence# 91093 first change number 6762363293401 next log first change 6762363625904 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:46,011] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91094_1014585229.arc thread# 1 sequence# 91094 first change number 6762363625904 next log first change 6762363855110 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:47,429] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91095_1014585229.arc thread# 1 sequence# 91095 first change number 6762363855110 next log first change 6762364169369 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:48,752] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:10:48,752] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:10:49,012] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91096_1014585229.arc thread# 1 sequence# 91096 first change number 6762364169369 next log first change 6762364364028 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:50,692] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91097_1014585229.arc thread# 1 sequence# 91097 first change number 6762364364028 next log first change 6762364600543 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:51,982] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91098_1014585229.arc thread# 1 sequence# 91098 first change number 6762364600543 next log first change 6762364841203 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:53,503] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91099_1014585229.arc thread# 1 sequence# 91099 first change number 6762364841203 next log first change 6762365069720 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:55,367] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91200_1014585229.arc thread# 1 sequence# 91200 first change number 6762400615006 next log first change 6762400930393 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:55,411] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:567) [2022-01-16 14:10:55,411] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] SQL errorCode = 0, SQL state = 'null' (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:570) [2022-01-16 14:10:55,412] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.run(OraCdcLogMinerWorkerThread.java:536) Caused by: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:148) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1048) at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1531) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1311) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3941) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010) at oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1StatementProxy$2oracle$1jdbc$1internal$1OracleCallableStatement$$$Proxy.execute(Unknown Source) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:219) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.next(OraCdcV$ArchivedLogImpl.java:125) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.run(OraCdcLogMinerWorkerThread.java:531) Caused by: Error : 1291, Position : 0, Sql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => :1 , ENDSCN => :2 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , OriginalSql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , Error Msg = ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637) ... 17 more (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:573) [2022-01-16 14:10:55,412] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Stopping oracdc logminer source task. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:687) [2022-01-16 14:10:56,213] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Saving oracdc state to /tmp/oracdc.state file... (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:737) [2022-01-16 14:10:56,239] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] oracdc state saved to /tmp/oracdc.state file, elapsed 26 ms (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:800) Exception in thread "OraCdcLogMinerWorkerThread-13796312078301893" org.apache.kafka.connect.errors.ConnectException: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.run(OraCdcLogMinerWorkerThread.java:579) Caused by: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.run(OraCdcLogMinerWorkerThread.java:536) Caused by: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:148) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1048) at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1531) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1311) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3941) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010) at oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1StatementProxy$2oracle$1jdbc$1internal$1OracleCallableStatement$$$Proxy.execute(Unknown Source) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:219) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.next(OraCdcV$ArchivedLogImpl.java:125) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.run(OraCdcLogMinerWorkerThread.java:531) Caused by: Error : 1291, Position : 0, Sql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => :1 , ENDSCN => :2 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , OriginalSql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , Error Msg = ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637) ... 17 more [2022-01-16 14:11:03,752] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:11:03,753] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:11:18,753] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:11:18,753] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:11:33,753] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:11:33,753] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:11:48,754] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:11:48,754] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:12:03,754] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-16 14:12:03,754] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-16 14:12:18,754] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-011-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430)

averemee-si commented 2 years ago

Hi Farooq,

Did you run RMAN command

crosscheck archivelog all

to mark all inaccessible logs as deleted?

Regards, Aleksei

averemee-si commented 2 years ago

Hi Farooq,

According to log last processed SEQUENCE# is 91200. What information present about SEQUENCE# = 91201 in V$ARCHIVED_LOG? Is file /archive/BREWIQFRMN/BREWIQFRMN_1_91201_1014585229.arc exist and accessible? What version of RDBMS you using?

Regards, Aleksei

FarooqWW commented 2 years ago

Yes we did following

crosscheck archivelog all; delete noprompt expired archivelog all;

Yes the file does exist

[oracle@BREWIQFRMN:] ls -ltr /archive/WWIQFRMN/WWIQFRMN_1_91201_1014585229.arc -rw-r----- 1 oracle dba 6216704 Jan 15 09:51 /archive/BREWIQFRMN/BREWIQFRMN_1_91201_1014585229.arc

` select name from v$archived_log where SEQUENCE#=91201;

/archive/BREWIQFRMN/BREWIQFRMN_1_91201_1014585229.arc `

FarooqWW commented 2 years ago

Here is the list of all archives available in v$archived_log view and present on the os.

SYS@BREWIQFRMN> select name,deleted,archived from v$archived_log where name like '/archive%';

/archive/BREWIQFRMN/BREWIQFRMN_1_91000_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91001_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91002_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91003_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91004_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91005_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91006_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91007_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91008_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91009_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91010_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91011_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91012_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91013_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91014_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91015_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91016_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91017_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91018_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91019_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91020_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91021_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91022_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91023_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91024_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91025_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91026_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91027_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91028_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91029_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91030_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91031_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91032_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91033_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91034_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91035_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91036_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91037_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91038_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91039_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91040_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91041_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91042_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91043_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91044_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91045_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91046_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91047_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91048_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91049_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91050_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91051_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91052_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91053_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91054_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91055_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91056_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91057_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91058_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91059_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91060_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91061_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91062_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91063_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91064_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91065_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91066_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91067_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91068_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91069_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91070_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91071_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91072_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91073_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91074_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91075_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91076_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91077_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91078_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91079_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91080_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91081_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91082_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91083_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91084_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91085_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91086_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91087_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91088_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91089_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91090_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91091_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91092_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91093_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91094_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91095_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91096_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91097_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91098_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91099_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91200_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91201_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91202_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91203_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91204_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91205_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91206_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91207_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91208_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91209_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91210_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91211_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91212_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91213_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91214_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91215_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91216_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91217_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91218_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91219_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91220_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91221_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91222_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91223_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91224_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91225_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91226_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91227_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91228_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91229_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91230_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91231_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91232_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91233_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91234_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91235_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91236_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91237_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91238_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91239_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91240_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91241_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91242_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91243_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91244_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91245_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91246_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91247_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91248_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91249_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91250_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91251_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91252_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91253_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91254_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91255_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91256_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91257_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91258_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91259_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91260_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91261_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91262_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91263_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91264_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91265_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91266_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91267_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91268_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91269_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91270_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91271_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91272_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91273_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91274_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91275_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91276_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91277_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91278_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91279_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91280_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91281_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91282_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91283_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91284_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91285_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91286_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91287_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91288_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91289_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91290_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91291_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91292_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91293_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91294_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91295_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91296_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91297_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91298_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91299_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91300_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91301_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91302_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91303_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91304_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91305_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91306_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91307_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91308_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91309_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91310_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91311_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91312_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91313_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91314_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91315_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91316_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91317_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91318_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91319_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91320_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91321_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91322_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91323_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91324_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91325_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91326_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91327_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91328_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91329_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91330_1014585229.arc NO YES /archive/BREWIQFRMN/BREWIQFRMN_1_91331_1014585229.arc NO YES

FarooqWW commented 2 years ago

DB Version:

SELECT BANNER_FULL FROM V$VERSION;

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.13.0.0.0

averemee-si commented 2 years ago

Thanks!

What version of RDBMS you are using?

From log: [2022-01-16 14:10:53,503] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91099_1014585229.arc thread# 1 sequence# 91099 first change number 6762364841203 next log first change 6762365069720 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:55,367] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91200_1014585229.arc thread# 1 sequence# 91200 first change number 6762400615006 next log first change 6762400930393 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 14:10:55,411] ERROR

Is it OK with 100 missed archived logs between SEQUENCE# 91099 and 91200?

Could you please try in SQL*Plus or other CLI (unfortunately I didn't know NEXT_CHANGE# for 91201 ...)

begin
  DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91200_1014585229.arc' , OPTIONS => DBMS_LOGMNR.NEW);
  DBMS_LOGMNR.START_LOGMNR( STARTSCN => 6762400615006 , ENDSCN => 6762365069720 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT);
end;
/

begin
  DBMS_LOGMNR.ADD_LOGFILE(LOGFILENAME => '/archive/BREWIQFRMN/BREWIQFRMN_1_91201_1014585229.arc' , OPTIONS => DBMS_LOGMNR.NEW);
  DBMS_LOGMNR.START_LOGMNR( STARTSCN => 6762365069720 , ENDSCN => <SEQ_91201_NEXT_CHANGE#> , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT);
end;
/

Regards, Aleksei

FarooqWW commented 2 years ago

Yes we are ok with missing sequences as this is nonprod instance. we are using Oracle 19c as I updated above. Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.13.0.0.0

FarooqWW commented 2 years ago

I executed above two add logfile blocks by getting the next_change# from v$archived_log. Do we need to perform any other action to let the connector fetch the data?

averemee-si commented 2 years ago

Hi Farooq,

Thanks!

If no errors from CLI with starting LogMiner for SEQUENCE# 91200 & 91201 could you please restart connector. I have only one idea about ORA-1291: add more output to the log if ORA-1291 occurs. Just in case: do you connecting to CDB$ROOT or to PDB?

Regards, Aleksei

FarooqWW commented 2 years ago

Aleksei, Its a non-cdb instance. regular RDBMS without multi-tenancy.

I restarted the connector however it failed at same step. It had started mining from archivelog 91000 and failed at 91200.

[2022-01-16 15:39:59,950] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91000_1014585229.arc thread# 1 sequence# 91000 first change number 6762323596064 next log first change 6762323874909 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174)

then failed at:

[2022-01-16 15:42:02,851] INFO [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91200_1014585229.arc thread# 1 sequence# 91200 first change number 6762400615006 next log first change 6762400930393 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:174) [2022-01-16 15:42:02,896] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:567) [2022-01-16 15:42:02,896] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] SQL errorCode = 0, SQL state = 'null' (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:570) [2022-01-16 15:42:02,897] ERROR [cluster4-BREWiqfrmn-a2-oracdc-011-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2

averemee-si commented 2 years ago

Hi Farooq,

I don't understand why the call works in SQL*Plus and doesn't work via JDBC... I will back to you shortly

Regards, Aleksei

averemee-si commented 2 years ago

Hi Farooq,

Could you please download updated connector from https://drive.google.com/file/d/1ijnXjPFrG2cXdoY7E0HtCxxpiG084bGs/view?usp=sharing and try using it. Additional output added for diagnostic ORA-1291 issue.

Regards, Aleksei

averemee-si commented 2 years ago

I have one idea but need time to check

Regards, Aleksei

FarooqWW commented 2 years ago

Sure I will have Roudy install this connector and then we will test. We might have to wait till Tuesday as monday is holiday in USA.

averemee-si commented 2 years ago

Happy Holidays!

averemee-si commented 2 years ago

Hi Farooq,

Connector with fix for ORA-1291 available at https://drive.google.com/file/d/1uqiA3UE3aVG5mRWUIs9L7SoJ9nH9xF76/view?usp=sharing. Please download it and retest issue.

Regards, Aleksei

FarooqWW commented 2 years ago

Hi There, So we upgraded the cluster to 0.9.9 and recreated the connector however it failed right after deployment. On DB side I had executed the crosskcheck archivelog all and delete expired archivelog all before deploying the connector. Also I had removed the "a2.first.change": "6762323596064", from the config which was added for testing.

Here is the complete log:

[2022-01-19 14:53:55,142] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Kafka version: 5.5.2-ccs (org.apache.kafka.common.utils.AppInfoParser:117) [2022-01-19 14:53:55,142] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Kafka commitId: 497d4b009a0ab25a (org.apache.kafka.common.utils.AppInfoParser:118) [2022-01-19 14:53:55,143] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Kafka startTimeMs: 1642604035142 (org.apache.kafka.common.utils.AppInfoParser:119) [2022-01-19 14:53:55,143] INFO [Worker clientId=connect-1, groupId=kafka-connect-cluster-oracle3-us-east-1-shared.aws] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1171) [2022-01-19 14:53:55,144] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Starting oracdc logminer source task (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:112) [2022-01-19 14:53:55,266] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] [Producer clientId=connector-producer-cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0] Cluster ID: REDACTED (org.apache.kafka.clients.Metadata:277) [2022-01-19 14:53:55,720] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Will start processing using stored persistent state file /tmp/oracdc.state dated 2022-01-19T14:25:34.27. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:248) [2022-01-19 14:53:55,722] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Restore persistent state 50 ms (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:323) [2022-01-19 14:53:55,724] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Stored state file /tmp/oracdc.state copied to /tmp/oracdc.state.1642604035722 (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:328) [2022-01-19 14:53:55,789] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Initializing oracdc logminer archivelog worker thread (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:121) [2022-01-19 14:53:56,020] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] LogMiner will start from SCN 6762365069698 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:102) [2022-01-19 14:53:56,020] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Mining database BREWIQFRMN_ORAHOST01 is in OPEN mode (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:209) [2022-01-19 14:53:56,020] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Same database will be used for dictionary query and mining (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:211) [2022-01-19 14:53:56,021] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] RowPrefetch size for accessing V$LOGMNR_CONTENTS set to 32. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:229) [2022-01-19 14:53:56,349] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Adding archived log /archive/BREWIQFRMN/BREWIQFRMN_1_91618_1014585229.arc thread# 1 sequence# 91618 first change number 6762563599428 next log first change 6762563735659 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:196) [2022-01-19 14:53:56,411] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Unable to execute begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; using STARTSCN=6762365069698 and ENDSCN=6762563735659 (eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl:247) [2022-01-19 14:53:56,412] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Unable to start logminer archivelog worker thread! (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:246) [2022-01-19 14:53:56,412] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:249) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.next(OraCdcV$ArchivedLogImpl.java:136) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.<init>(OraCdcLogMinerWorkerThread.java:232) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:518) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:148) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1048) at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1531) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1311) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3941) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010) at oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1StatementProxy$2oracle$1jdbc$1internal$1OracleCallableStatement$$$Proxy.execute(Unknown Source) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:244) ... 11 more Caused by: Error : 1291, Position : 0, Sql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => :1 , ENDSCN => :2 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , OriginalSql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , Error Msg = ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637) ... 26 more (eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread:247) [2022-01-19 14:53:56,412] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Unable to start oracdc logminer task! (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:540) [2022-01-19 14:53:56,413] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.<init>(OraCdcLogMinerWorkerThread.java:248) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:518) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:249) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.next(OraCdcV$ArchivedLogImpl.java:136) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.<init>(OraCdcLogMinerWorkerThread.java:232) ... 9 more Caused by: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:148) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1048) at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1531) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1311) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3941) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010) at oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1StatementProxy$2oracle$1jdbc$1internal$1OracleCallableStatement$$$Proxy.execute(Unknown Source) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:244) ... 11 more Caused by: Error : 1291, Position : 0, Sql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => :1 , ENDSCN => :2 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , OriginalSql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , Error Msg = ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637) ... 26 more (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:541) [2022-01-19 14:53:56,413] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-19 14:53:56,413] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447) [2022-01-19 14:53:56,413] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:187) org.apache.kafka.connect.errors.ConnectException: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:542) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.<init>(OraCdcLogMinerWorkerThread.java:248) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:518) ... 8 more Caused by: java.sql.SQLException: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:249) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.next(OraCdcV$ArchivedLogImpl.java:136) at eu.solutions.a2.cdc.oracle.OraCdcLogMinerWorkerThread.<init>(OraCdcLogMinerWorkerThread.java:232) ... 9 more Caused by: java.sql.SQLException: ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630) at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564) at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151) at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771) at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299) at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498) at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:148) at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1048) at oracle.jdbc.driver.OracleStatement.executeSQLStatement(OracleStatement.java:1531) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1311) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746) at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3941) at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4279) at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1010) at oracle.ucp.jdbc.proxy.oracle$1ucp$1jdbc$1proxy$1oracle$1StatementProxy$2oracle$1jdbc$1internal$1OracleCallableStatement$$$Proxy.execute(Unknown Source) at eu.solutions.a2.cdc.oracle.OraCdcV$ArchivedLogImpl.start(OraCdcV$ArchivedLogImpl.java:244) ... 11 more Caused by: Error : 1291, Position : 0, Sql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => :1 , ENDSCN => :2 , OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , OriginalSql = begin DBMS_LOGMNR.START_LOGMNR( STARTSCN => ?, ENDSCN => ?, OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); end; , Error Msg = ORA-01291: fichier journal manquant ORA-06512: à "SYS.DBMS_LOGMNR", ligne 72 ORA-06512: à ligne 2 at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637) ... 26 more [2022-01-19 14:53:56,413] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:188) [2022-01-19 14:53:56,414] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Stopping oracdc logminer source task. (eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask:698) [2022-01-19 14:53:56,414] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] [Producer clientId=connector-producer-cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer:1182) [2022-01-19 14:54:10,144] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:430) [2022-01-19 14:54:10,144] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0|offsets] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:447)

averemee-si commented 2 years ago

Hi Farooq

Could you please run on server with connector

rm /tmp/oracdc.state

and restart connector.

If this does not helps:

  1. Does it run from from SQL*Plus: 
    begin
DBMS_LOGMNR.START_LOGMNR(
STARTSCN => 6762365069698,
ENDSCN => 6762563735659,
OPTIONS => DBMS_LOGMNR.SKIP_CORRUPTION + DBMS_LOGMNR.NO_SQL_DELIMITER + DBMS_LOGMNR.NO_ROWID_IN_STMT); 
end;
  2. Please also provide output for 
    select FIRST_CHANGE#, NEXT_CHANGE# from V$ARCHIVED_LOG where SEQUENCE# = 91618;

Regards, Aleksei

FarooqWW commented 2 years ago

Sure I will check and update you. Has this config parameter changed as I started getting this error too. a2.include

connector showing this message too.

"trace": "org.apache.kafka.connect.errors.ConnectException: Please check value of a2.include parameter or remove it from configuration!\n\tat eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.s

FarooqWW commented 2 years ago

Aleksei, I restarted the cluster and removed the schema name from a2.include and restarted the connector as well. Now I am getting new error.

[2022-01-19 16:03:58,924] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Process id: 20 :: Chronicle Queue (5.21.93) (net.openhft.chronicle.core.internal.announcer.InternalAnnouncer:56) [2022-01-19 16:03:58,925] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Analytics: Chronicle Queue reports usage statistics. Learn more or turn off: https://github.com/OpenHFT/Chronicle-Queue/blob/master/DISCLAIMER.adoc (net.openhft.chronicle.core.internal.announcer.InternalAnnouncer:56) Client id file not present: /root/.chronicle.analytics.client.id java.nio.file.NoSuchFileException: /root/.chronicle.analytics.client.id [2022-01-19 16:03:58,967] INFO [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Took 3.445 ms to pollDiskSpace for /tmp/07002100D4780F00.5505880511385435294 (net.openhft.chronicle.threads.DiskSpaceMonitor:56) [2022-01-19 16:04:00,178] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] Error encountered in task cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0. Executing stage 'VALUE_CONVERTER' with class 'io.confluent.connect.avro.AvroConverter'. (org.apache.kafka.connect.runtime.errors.LogReporter:62) org.apache.kafka.connect.errors.DataException: Failed to serialize Avro data from topic BREWiqfrmn_a2-oracdc-BREWFR.EXAMPLE_TEST_ORA_CDC : at io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:91) at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63) at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:301) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104) at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:301) at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:327) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:251) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: ["null",{"type":"record","name":"Value","namespace":"BREWFR.EXAMPLE_TEST_ORA_CDC","fields":[{"name":"TAB_ID","type":{"type":"bytes","scale":0,"precision":64,"connect.version":1,"connect.parameters":{"scale":"0"},"connect.name":"org.apache.kafka.connect.data.Decimal","logicalType":"decimal"}},{"name":"ORDER_ID","type":"string"},{"name":"SUBS_NOTIFY_TYPE","type":"int"},{"name":"ACKNOWLEDGED_STATE","type":["null","int"],"default":null},{"name":"MEMBER_NO","type":"string"},{"name":"SUBSCRIPTION_ID","type":["null","string"],"default":null},{"name":"LATEST_RECEIPT","type":["null","string"],"default":null}],"connect.version":1,"connect.name":"BREWFR.EXAMPLE_TEST_ORA_CDC.Value"}] Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292) at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:495) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:486) at io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:459) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:206) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:268) at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:244) at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:75) at io.confluent.connect.avro.AvroConverter$Serializer.serialize(AvroConverter.java:143) at io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:84) at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63) at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:301) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162) at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104) at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:301) at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:327) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:251) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

averemee-si commented 2 years ago

Cool!!!

I have one more issue to fix, thanks!!!

About last error:

Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unauthorized; error code: 401 at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)

schema registry auth missed. For Confluent cloud you need to set (Ref.: https://docs.confluent.io/cloud/current/cp-component/streams-cloud-config.html)

schema.registry.basic.auth.user.info="<SCHEMA_REGISTRY_API_KEY>:<SCHEMA_REGISTRY_API_SECRET>"

Hope this helps

Regards, Aleksei

FarooqWW commented 2 years ago

Aleksei, One more problem i am facing is when I edit a connector, i start getting following error. I have to restart cluster to reset everything. can you help fix this JMX issue?

{ "name": "cluster4-BREWiqfrmn-a2-oracdc-012-nonprod", "connector": { "state": "RUNNING", "worker_id": "0.0.0.0:8083" }, "tasks": [ { "id": 0, "state": "FAILED", "worker_id": "0.0.0.0:8083", "trace": "org.apache.kafka.connect.errors.ConnectException: javax.management.InstanceAlreadyExistsException: eu.solutions.a2.oracdc:type=LogMiner-metrics,name=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod,database=BREWIQFRMN_ORAHOST1\n\tat eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmtBase.<init>(OraCdcLogMinerMgmtBase.java:79)\n\tat eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmt.<init>(OraCdcLogMinerMgmt.java:58)\n\tat eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:176)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: javax.management.InstanceAlreadyExistsException: eu.solutions.a2.oracdc:type=LogMiner-metrics,name=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod,database=BREWIQFRMN_ORAHOST1\n\tat com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)\n\tat eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmtBase.<init>(OraCdcLogMinerMgmtBase.java:75)\n\t... 10 more\n" } ], "type": "source" }

averemee-si commented 2 years ago

OK,

It's easy with JMX. I will quickly prepare a fix and notify you

Regards, Aleksei

FarooqWW commented 2 years ago

Please help me fix this warning too.

"trace": "org.apache.kafka.connect.errors.ConnectException: Please check value of a2.include parameter or remove it from configuration!\n\tat eu.solutions.a2.cdc.oracle.O

The parameter was working fine before the 0.9.9 upgrade.

"a2.include": "BEWWFR. EXAMPLE_TEST_ORA_CDC",

rveve commented 2 years ago

Also, regarding the JMX issue, I want to note, even you disable jmx at the JVM level, you are still getting this error in the kafka connect log but not in the connector status Task trace, that is better.

org.apache.kafka.connect.errors.ConnectException: javax.management.InstanceAlreadyExistsException: eu.solutions.a2.oracdc:type=LogMiner-metrics,name=CL4_BREWFR-a2-oracdc-001-nonprod,database=CL4_BREWFR
    at eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmtBase.<init>(OraCdcLogMinerMgmtBase.java:79)
    at eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmt.<init>(OraCdcLogMinerMgmt.java:58)
    at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:176)
    at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219)
    at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
    at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.management.InstanceAlreadyExistsException: eu.solutions.a2.oracdc:type=LogMiner-metrics,name=CL4_BREWFR-a2-oracdc-001-nonprod,database=CL$_BREWFR
    at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
    at eu.solutions.a2.cdc.oracle.jmx.OraCdcLogMinerMgmtBase.<init>(OraCdcLogMinerMgmtBase.java:75)
    ... 10 more

Above connect cluster jvm process was started without jmx enabled.

averemee-si commented 2 years ago

OK, working on it. Hope that fixes will be ready soon. JMX is very important, it contains very useful metrics - https://github.com/averemee-si/oracdc/blob/master/doc/LOGMINER-METRICS.md

Regards, Aleksei

averemee-si commented 2 years ago

Colleagues, Just for information - in log exist /tmp/07002100D4780F00.5505880511385435294 this is memory mapped file which contains Oracle transaction. oracdc uses LogMiner only for reading and performs transaction parsing at connector side to minimize side effect of CDC for RDBMS.

Regards, Aleksei

averemee-si commented 2 years ago

Updated connector available at - https://drive.google.com/file/d/1-NX2lGkZ9aLejyNUonmYiH_HtkF7pH06/view?usp=sharing

Regards, Aleksei

FarooqWW commented 2 years ago

Aleksei, Thanks for your continuous help. we have fixed a lot of issues. We have installed latest connector and restarted the cluster. Now we are facing following challenge. I deployed two connectors (each one pointing to different DB on different host) they both failed with same errors.

[2022-01-20 01:12:08,378] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:187) org.apache.kafka.connect.errors.ConnectException: Please check value of a2.include parameter or remove it from configuration! at eu.solutions.a2.cdc.oracle.OraCdcLogMinerTask.start(OraCdcLogMinerTask.java:404) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2022-01-20 01:12:08,378] ERROR [cluster4-BREWiqfrmn-a2-oracdc-012-nonprod|task-0] WorkerSourceTask{id=cluster4-BREWiqfrmn-a2-oracdc-012-nonprod-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:188)

averemee-si commented 2 years ago

Hi Farooq,

What value is a2.include set to? Is it possible to receive by e-mail (oracle@a2-solutions.eu) a complete log from both servers including parameter values?

Regards, Aleksei

FarooqWW commented 2 years ago

Aleksei, Good Morning, We haven't changed this parameter value which was working fine before 0.9.9 upgrade.

The parameter was working fine before the 0.9.9 upgrade.

"a2.include": "BEWWFR. EXAMPLE_TEST_ORA_CDC",

Interestingly if I remove the schema name from this, the connector starts working however its not loading anything. I will send you complete log shortly.

FarooqWW commented 2 years ago

Aleksei, I think I found the reason for failure. We are using the default parameter for state file /tmp/oracdc.state So both connectors fighting to acquire the ownership and hence giving all these weird errors. I paused one of the connector and the other one started working. As soon as I resumed the first one, the second one also failed. Let me change that and test.