search

apache / flink-cdc

Flink CDC is a streaming data integration tool
https://nightlies.apache.org/flink/flink-cdc-docs-stable
Apache License 2.0
5.71k stars 1.94k forks source link

[BUG] ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access #1255

Closed atovk closed 2 years ago

atovk commented 2 years ago

Describe the bug(Please use English) when flink cdc oracle data to sink print, has some data sinked print, but has fatal error!!!

this flink job failure!!!!

Environment :

To Reproduce Steps to reproduce the behavior:

  1. Thes test data :
  2. The test code :
  3. The key error :

2022-06-08 17:49:40,618 INFO io.debezium.pipeline.ChangeEventSourceCoordinator [] - Metrics registered 2022-06-08 17:49:40,619 INFO io.debezium.pipeline.ChangeEventSourceCoordinator [] - Context created 2022-06-08 17:49:40,625 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 1 - Preparing 2022-06-08 17:49:40,625 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 2 - Determining captured tables 2022-06-08 17:49:41,046 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 3 - Locking captured tables [orcl01.DCSDBA.MOVE_TASK] 2022-06-08 17:49:41,111 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 4 - Determining snapshot offset 2022-06-08 17:49:41,220 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 5 - Reading structure of captured tables 2022-06-08 17:49:42,390 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 6 - Persisting schema history 2022-06-08 17:49:44,894 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshot step 7 - Snapshotting data 2022-06-08 17:49:44,895 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Snapshotting contents of 1 tables while still in transaction 2022-06-08 17:49:44,895 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Exporting data from table 'orcl01.DCSDBA.MOVE_TASK' (1 of 1 tables) 2022-06-08 17:49:44,896 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - For table 'orcl01.DCSDBA.MOVE_TASK' using select statement: 'SELECT * FROM "DCSDBA"."MOVE_TASK" AS OF SCN 3870785327275' 2022-06-08 17:49:45,104 INFO com.ververica.cdc.debezium.internal.DebeziumChangeFetcher [] - Database snapshot phase can't perform checkpoint, acquired Checkpoint lock. 2022-06-08 17:49:45,215 INFO io.debezium.connector.common.BaseSourceTask [] - 1998 records sent during previous 00:00:05.057, last recorded offset: {snapshot=true, scn=3870785327275, snapshot_completed=false} 2022-06-08 17:49:50,684 INFO io.debezium.relational.RelationalSnapshotChangeEventSource [] - Finished exporting 97708 records for table 'orcl01.DCSDBA.MOVE_TASK'; total duration '00:00:05.789' 2022-06-08 17:49:50,699 INFO io.debezium.pipeline.source.AbstractSnapshotChangeEventSource [] - Snapshot - Final stage 2022-06-08 17:49:50,700 INFO io.debezium.pipeline.ChangeEventSourceCoordinator [] - Snapshot ended with SnapshotResult [status=COMPLETED, offset=OracleOffsetContext [scn=3870785327275]] 2022-06-08 17:49:50,705 INFO io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics [] - Connected metrics set to 'true' 2022-06-08 17:49:50,706 INFO io.debezium.pipeline.ChangeEventSourceCoordinator [] - Starting streaming 2022-06-08 17:49:50,929 ERROR io.debezium.connector.oracle.logminer.LogMinerHelper [] - Mining session stopped due to the java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access

2022-06-08 17:49:50,929 ERROR io.debezium.pipeline.ErrorHandler [] - Producer failure java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:550) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:268) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:265) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:86) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:965) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1205) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3666) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1358) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3778) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4251) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1081) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:693) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.createFlushTable(LogMinerHelper.java:122) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:124) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:152) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:119) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_251]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

Caused by: oracle.jdbc.OracleDatabaseException: ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:513) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
... 24 more

2022-06-08 17:49:50,941 INFO io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource [] - startScn=3870785327275, endScn=null, offsetContext.getScn()=3870785327275 2022-06-08 17:49:50,942 INFO io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource [] - Transactional buffer dump: 2022-06-08 17:49:50,943 INFO io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource [] - Streaming metrics dump: OracleStreamingChangeEventSourceMetrics{currentScn=null, oldestScn=null, committedScn=null, offsetScn=null, logMinerQueryCount=0, totalProcessedRows=0, totalCapturedDmlCount=0, totalDurationOfFetchingQuery=PT0S, lastCapturedDmlCount=0, lastDurationOfFetchingQuery=PT0S, maxCapturedDmlCount=0, maxDurationOfFetchingQuery=PT0S, totalBatchProcessingDuration=PT0S, lastBatchProcessingDuration=PT0S, maxBatchProcessingDuration=PT0S, maxBatchProcessingThroughput=0, currentLogFileName=null, minLogFilesMined=0, maxLogFilesMined=0, redoLogStatus=null, switchCounter=0, batchSize=20000, millisecondToSleepBetweenMiningQuery=1000, recordMiningHistory=false, hoursToKeepTransaction=0, networkConnectionProblemsCounter0, batchSizeDefault=20000, batchSizeMin=1000, batchSizeMax=100000, sleepTimeDefault=1000, sleepTimeMin=0, sleepTimeMax=3000, sleepTimeIncrement=200, totalParseTime=PT0S, totalStartLogMiningSessionDuration=PT0S, lastStartLogMiningSessionDuration=PT0S, maxStartLogMiningSessionDuration=PT0S, totalProcessTime=PT0S, minBatchProcessTime=PT0S, maxBatchProcessTime=PT0S, totalResultSetNextTime=PT0S, lagFromTheSource=DurationPT0S, maxLagFromTheSourceDuration=PT0S, minLagFromTheSourceDuration=PT0S, lastCommitDuration=PT0S, maxCommitDuration=PT0S, activeTransactions=0, rolledBackTransactions=0, committedTransactions=0, abandonedTransactionIds=[], rolledbackTransactionIds=[], registeredDmlCount=0, committedDmlCount=0, errorCount=1, warningCount=0, scnFreezeCount=0} 2022-06-08 17:49:50,943 INFO io.debezium.pipeline.ChangeEventSourceCoordinator [] - Finished streaming 2022-06-08 17:49:50,943 INFO io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics [] - Connected metrics set to 'false' 2022-06-08 17:49:51,230 WARN io.debezium.connector.common.BaseSourceTask [] - Going to restart connector after 10 sec. after a retriable exception 2022-06-08 17:49:51,231 WARN io.debezium.pipeline.ChangeEventSourceCoordinator [] - Unable to unregister the MBean 'debezium.oracle:type=connector-metrics,context=streaming,server=oracle_logminer': debezium.oracle:type=connector-metrics,context=streaming,server=oracle_logminer 2022-06-08 17:49:51,256 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed 2022-06-08 17:49:51,257 INFO io.debezium.embedded.EmbeddedEngine [] - Retrieable exception thrown, connector will be restarted com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:38) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:208) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:152) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:119) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_251] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_251] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251] Caused by: java.sql.SQLException: ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:550) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:268) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:265) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:86) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:965) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1205) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3666) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1358) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3778) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4251) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1081) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:693) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.createFlushTable(LogMinerHelper.java:122) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:124) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
... 7 more

Caused by: oracle.jdbc.OracleDatabaseException: ORA-00604: error occurred at recursive SQL level 1 ORA-16000: database open for read-only access

at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:513) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:550) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:268) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:265) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:86) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:965) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1205) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3666) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1358) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3778) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4251) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1081) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.executeCallableStatement(LogMinerHelper.java:693) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerHelper.createFlushTable(LogMinerHelper.java:122) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
at io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:124) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT]
... 7 more

2022-06-08 17:49:51,258 INFO io.debezium.connector.common.BaseSourceTask [] - Awaiting end of restart backoff period after a retriable error 2022-06-08 17:49:51,438 INFO com.ververica.cdc.debezium.internal.DebeziumChangeFetcher [] - Received record from streaming binlog phase, released checkpoint lock. 2022-06-08 17:49:53,259 INFO io.debezium.connector.common.BaseSourceTask [] - Awaiting end of restart backoff period after a retriable error 2022-06-08 17:49:55,260 INFO io.debezium.connector.common.BaseSourceTask [] - Awaiting end of restart backoff period after a retriable error 2022-06-08 17:49:57,260 INFO io.debezium.connector.common.BaseSourceTask [] - Awaiting end of restart backoff period after a retriable error 2022-06-08 17:49:59,261 INFO io.debezium.connector.common.BaseSourceTask [] - Awaiting end of restart backoff period after a retriable error 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - Starting OracleConnectorTask with configuration: 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - connector.class = io.debezium.connector.oracle.OracleConnector 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.user = bd_cdc_flink 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.dbname = orcl01 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - offset.storage = com.ververica.cdc.debezium.internal.FlinkOffsetBackingStore 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.tablename.case.insensitive = false 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.server.name = oracle_logminer 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - log.mining.strategy = online_catalog 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - include.schema.changes = false 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.port = 1521 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - schema.whitelist = DCSDBA 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - offset.flush.interval.ms = 9223372036854775807 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - tombstones.on.delete = false 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - log.mining.continuous.mine = true 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.hostname = 192.168.0.88 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.password = **** 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - name = engine 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.history.skip.unparseable.ddl = true 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - table.include.list = DCSDBA.MOVE_TASK 2022-06-08 17:50:01,262 INFO io.debezium.connector.common.BaseSourceTask [] - database.history.instance.name = 47b1565e-e7bc-4f7b-b49f-6d406e357385 2022-06-08 17:50:01,263 INFO io.debezium.connector.common.BaseSourceTask [] - snapshot.mode = initial 2022-06-08 17:50:01,263 INFO io.debezium.connector.common.BaseSourceTask [] - database.history = com.ververica.cdc.debezium.internal.FlinkDatabaseSchemaHistory 2022-06-08 17:50:01,263 WARN io.debezium.connector.oracle.OracleConnectorConfig [] - The option 'database.tablename.case.insensitive' is deprecated and will be removed in the future. 2022-06-08 17:50:01,306 INFO io.debezium.connector.oracle.OracleConnection [] - Database Version: Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production 2022-06-08 17:50:01,308 INFO io.debezium.connector.common.BaseSourceTask [] - Stopping down connector 2022-06-08 17:50:01,308 WARN io.debezium.pipeline.ChangeEventSourceCoordinator [] - Unable to unregister the MBean 'debezium.oracle:type=connector-metrics,context=snapshot,server=oracle_logminer': debezium.oracle:type=connector-metrics,context=snapshot,server=oracle_logminer 2022-06-08 17:50:01,309 WARN io.debezium.pipeline.ChangeEventSourceCoordinator [] - Unable to unregister the MBean 'debezium.oracle:type=connector-metrics,context=streaming,server=oracle_logminer': debezium.oracle:type=connector-metrics,context=streaming,server=oracle_logminer 2022-06-08 17:50:01,312 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed 2022-06-08 17:50:01,313 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to unregister the MBean 'debezium.oracle:type=connector-metrics,context=schema-history,server=oracle_logminer': debezium.oracle:type=connector-metrics,context=schema-history,server=oracle_logminer 2022-06-08 17:50:01,316 ERROR com.ververica.cdc.debezium.internal.Handover [] - Reporting error: java.lang.IllegalStateException: Retrieve schema history failed, the schema records for engine 47b1565e-e7bc-4f7b-b49f-6d406e357385 has been removed, this might because the debezium engine has been shutdown due to other errors. at com.ververica.cdc.debezium.utils.DatabaseHistoryUtil.retrieveHistory(DatabaseHistoryUtil.java:77) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at com.ververica.cdc.debezium.internal.FlinkDatabaseSchemaHistory.configure(FlinkDatabaseSchemaHistory.java:82) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig.getDatabaseHistory(HistorizedRelationalDatabaseConnectorConfig.java:105) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.relational.HistorizedRelationalDatabaseSchema.(HistorizedRelationalDatabaseSchema.java:39) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.oracle.OracleDatabaseSchema.(OracleDatabaseSchema.java:35) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.oracle.OracleConnectorTask.start(OracleConnectorTask.java:54) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:207) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:148) ~[flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:788) [flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188) [flink-sql-connector-oracle-cdc-2.2-apply.jar:2.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251] 2022-06-08 17:50:01,320 INFO io.debezium.embedded.EmbeddedEngine [] - Stopping the embedded engine 2022-06-08 17:50:01,323 INFO io.debezium.embedded.EmbeddedEngine [] - Stopping the embedded engine 2022-06-08 17:50:01,325 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: TableSourceScan(table=[[default_catalog, default_database, move_task_flink]], fields=[KEY, FIRST_KEY, TASK_TYPE, TASK_ID, LINE_ID, CLIENT_ID, SKU_ID, CONFIG_ID, DESCRIPTION, TAG_ID, OLD_TAG_ID, CUSTOMER_ID, ORIGIN_ID, CONDITION_ID, QTY_TO_MOVE, OLD_QTY_TO_MOVE, SITE_ID, FROM_LOC_ID, OLD_FROM_LOC_ID, TO_LOC_ID, OLD_TO_LOC_ID, FINAL_LOC_ID, OWNER_ID, SEQUENCE, STATUS, LIST_ID, DSTAMP, START_DSTAMP, FINISH_DSTAMP, ORIGINAL_DSTAMP, PRIORITY, CONSOL_LINK, FACE_TYPE, FACE_KEY, WORK_ZONE, WORK_GROUP, CONSIGNMENT, BOL_ID, REASON_CODE, CONTAINER_ID, TO_CONTAINER_ID, PALLET_ID, TO_PALLET_ID, TO_PALLET_CONFIG, TO_PALLET_VOLUME, TO_PALLET_HEIGHT, TO_PALLET_DEPTH, TO_PALLET_WIDTH, TO_PALLET_WEIGHT, PALLET_GROUPED, PALLET_CONFIG, PALLET_VOLUME, PALLET_HEIGHT, PALLET_DEPTH, PALLET_WIDTH, PALLET_WEIGHT, USER_ID, STATION_ID, SESSION_TYPE, SUMMARY_RECORD, REPACK, KIT_SKU_ID, KIT_LINE_ID, KIT_RATIO, KIT_LINK, DUE_TYPE, DUE_TASK_ID, DUE_LINE_ID, TRAILER_POSITION, CONSOLIDATED_TASK, DISALLOW_TAG_SWAP, CE_UNDER_BOND, INCREMENT_TIME, ESTIMATED_TIME, UPLOADED_LABOR, PRINT_LABEL_ID, PRINT_LABEL, OLD_STATUS, REPACK_QC_DONE, OLD_TASK_ID, CATCH_WEIGHT, MOVED_LOCK_STATUS, PICK_REALLOC_FLAG, STAGE_ROUTE_ID, STAGE_ROUTE_SEQUENCE, LABELLING, PF_CONSOL_LINK, INV_KEY, FIRST_PICK, SERIAL_NUMBER, LABEL_EXCEPTIONED, SHIPMENT_GROUP, DECONSOLIDATE, KIT_PLAN_ID, PLAN_SEQUENCE, TO_CONTAINER_CONFIG, CONTAINER_CONFIG, CE_ROTATION_ID, CE_AVAIL_STATUS, RDT_USER_MODE, CONSOL_RUN_NUMBIG, LABOR_ASSIGNMENT]) -> NotNullEnforcer(fields=[KEY, FIRST_KEY, TASK_TYPE, TASK_ID, SKU_ID, QTY_TO_MOVE, OLD_QTY_TO_MOVE, SITE_ID, STATUS, START_DSTAMP, FINISH_DSTAMP, ORIGINAL_DSTAMP, PRIORITY]) -> Sink: Sink(table=[default_catalog.default_database.target_move_task], fields=[KEY, FIRST_KEY, TASK_TYPE, TASK_ID, LINE_ID, CLIENT_ID, SKU_ID, CONFIG_ID, DESCRIPTION, TAG_ID, OLD_TAG_ID, CUSTOMER_ID, ORIGIN_ID, CONDITION_ID, QTY_TO_MOVE, OLD_QTY_TO_MOVE, SITE_ID, FROM_LOC_ID, OLD_FROM_LOC_ID, TO_LOC_ID, OLD_TO_LOC_ID, FINAL_LOC_ID, OWNER_ID, SEQUENCE, STATUS, LIST_ID, DSTAMP, START_DSTAMP, FINISH_DSTAMP, ORIGINAL_DSTAMP, PRIORITY, CONSOL_LINK, FACE_TYPE, FACE_KEY, WORK_ZONE, WORK_GROUP, CONSIGNMENT, BOL_ID, REASON_CODE, CONTAINER_ID, TO_CONTAINER_ID, PALLET_ID, TO_PALLET_ID, TO_PALLET_CONFIG, TO_PALLET_VOLUME, TO_PALLET_HEIGHT, TO_PALLET_DEPTH, TO_PALLET_WIDTH, TO_PALLET_WEIGHT, PALLET_GROUPED, PALLET_CONFIG, PALLET_VOLUME, PALLET_HEIGHT, PALLET_DEPTH, PALLET_WIDTH, PALLET_WEIGHT, USER_ID, STATION_ID, SESSION_TYPE, SUMMARY_RECORD, REPACK, KIT_SKU_ID, KIT_LINE_ID, KIT_RATIO, KIT_LINK, DUE_TYPE, DUE_TASK_ID, DUE_LINE_ID, TRAILER_POSITION, CONSOLIDATED_TASK, DISALLOW_TAG_SWAP, CE_UNDER_BOND, INCREMENT_TIME, ESTIMATED_TIME, UPLOADED_LABOR, PRINT_LABEL_ID, PRINT_LABEL, OLD_STATUS, REPACK_QC_DONE, OLD_TASK_ID, CATCH_WEIGHT, MOVED_LOCK_STATUS, PICK_REALLOC_FLAG, STAGE_ROUTE_ID, STAGE_ROUTE_SEQUENCE, LABELLING, PF_CONSOL_LINK, INV_KEY, FIRST_PICK, SERIAL_NUMBER, LABEL_EXCEPTIONED, SHIPMENT_GROUP, DECONSOLIDATE, KIT_PLAN_ID, PLAN_SEQUENCE, TO_CONTAINER_CONFIG, CONTAINER_CONFIG, CE_ROTATION_ID, CE_AVAIL_STATUS, RDT_USER_MODE, CONSOL_RUN_NUMBIG, LABOR_ASSIGNMENT]) (1/1)#0 (62a19159d33e9848678e25e7345fbdbd) switched from RUNNING to FAILED with failure cause: java.lang.IllegalStateException: Retrieve schema history failed, the schema records for engine 47b1565e-e7bc-4f7b-b49f-6d406e357385 has been removed, this might because the debezium engine has been shutdown due to other errors. at com.ververica.cdc.debezium.utils.DatabaseHistoryUtil.retrieveHistory(DatabaseHistoryUtil.java:77) at com.ververica.cdc.debezium.internal.FlinkDatabaseSchemaHistory.configure(FlinkDatabaseSchemaHistory.java:82) at io.debezium.relational.HistorizedRelationalDatabaseConnectorConfig.getDatabaseHistory(HistorizedRelationalDatabaseConnectorConfig.java:105) at io.debezium.relational.HistorizedRelationalDatabaseSchema.(HistorizedRelationalDatabaseSchema.java:39) at io.debezium.connector.oracle.OracleDatabaseSchema.(OracleDatabaseSchema.java:35) at io.debezium.connector.oracle.OracleConnectorTask.start(OracleConnectorTask.java:54) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:207) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:148) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:788) at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

2022-06-08 17:50:01,326 INFO io.debezium.embedded.EmbeddedEngine [] - Stopping the embedded [engine]



**Additional Description**
[flink-oracle-11g-cdc-error.log](https://github.com/ververica/flink-cdc-connectors/files/8860802/flink-oracle-11g-cdc-error.log)

full taskmanager log:
atovk commented 2 years ago

@leonardBang This issure seems common, and the bug is so weird 😂

atovk commented 2 years ago

this is oracle table with config: 

with(
    'connector' = 'oracle-cdc',
    'hostname' = '192.168.0.1',
    'port' = '1521',
    'username' = 'cdc_flink', 
    'password' = '12312312',
    'database-name' = 'oracl',
    'schema-name' = 'DCSDBA',  
    'table-name' = 'MOVE_TASK',
    'debezium.database.tablename.case.insensitive' = 'false',
    'debezium.log.mining.strategy'='online_catalog',
    'debezium.log.mining.continuous.mine'='true'
)
atovk commented 2 years ago

Found an issure of Debezium : Debezium issure can we set oracle: oracle.jdbc.timezoneAsRegion=false ?

atovk commented 2 years ago

Debezium oracle cdc unsupport read-only sdandby

Dsrong commented 1 year ago

Debezium oracle cdc unsupport read-only sdandby

how to slove it? I have same issue。now i can not submit application,All the application erroe message is "Retrieve schema history failed, the schema records for engine 47b1565e-e7bc-4f7b-b49f-6d406e357385 has been removed "