DipalPrajapati
commented
3 weeks ago 2024-07-09 13:46:26,318 INFO [org.apa.ice.BaseMetastoreTableOperations] (IcebergOffsetBackingStore-1) Successfully committed to table company.debezium_offset_storage_custom_table in 926 ms
2024-07-09 13:46:26,530 INFO [org.apa.ice.BaseMetastoreTableOperations] (IcebergOffsetBackingStore-1) Refreshing table metadata from new version: s3://iceberg-cdc-table/warehouse/company/debezium_offset_storage_custom_table/metadata/00007-67efc028-b29a-435d-8bf1-6e7507d63491.metadata.json
2024-07-09 13:46:27,044 INFO [com.git.shy.mys.bin.BinaryLogClient] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) SSL enabled
2024-07-09 13:46:29,157 INFO [com.git.shy.mys.bin.BinaryLogClient] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Connected to company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306 at mysql-bin-changelog.000002/39428 (sid:1234, cid:1221)
2024-07-09 13:46:29,158 INFO [io.deb.con.bin.BinlogStreamingChangeEventSource] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Connected to binlog at company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306, starting at BinlogOffsetContext{sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=BinlogSourceInfo{currentGtid='null', currentBinlogFilename='mysql-bin-changelog.000002', currentBinlogPosition=39428, currentRowNumber=0, serverId=0, sourceTime=2024-07-09T08:16:15.501Z, threadId=-1, currentQuery='null', tableIds=[company.iceberg_namespace_properties], databaseName='company'}, snapshotCompleted=true, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet='null', currentGtidSet='null', restartBinlogFilename='mysql-bin-changelog.000002', restartBinlogPosition=39428, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId='null', incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]}
2024-07-09 13:46:29,158 INFO [io.deb.con.bin.BinlogStreamingChangeEventSource] (debezium-mysqlconnector-dbz_-change-event-source-coordinator) Waiting for keepalive thread to start
2024-07-09 13:46:29,159 INFO [io.deb.uti.Threads] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Creating thread debezium-mysqlconnector-dbz_-binlog-client
2024-07-09 13:46:29,260 INFO [io.deb.con.bin.BinlogStreamingChangeEventSource] (debezium-mysqlconnector-dbz_-change-event-source-coordinator) Keepalive thread is running
] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Received query command: Event{header=EventHeaderV4{timestamp=1720517320000, eventType=QUERY, serverId=28029116, headerLength=19, dataLength=59, nextPosition=123565, flags=8}, data=QueryEventData{threadId=1184, executionTime=0, errorCode=0, database='company', sql='BEGIN'}}
2024-07-09 15:04:53,242 DEBUG [io.deb.con.bas.ChangeEventQueue] (pool-8-thread-1) no records available or batch size not reached yet, sleeping a bit...
2024-07-09 15:04:53,746 DEBUG [io.deb.con.bas.ChangeEventQueue] (pool-8-thread-1) checking for more records...
2024-07-09 15:04:53,746 DEBUG [io.deb.con.bas.ChangeEventQueue] (pool-8-thread-1) polling records...
2024-07-09 15:04:53,746 DEBUG [io.deb.con.bas.ChangeEventQueue] (pool-8-thread-1) no records available or batch size not reached yet, sleeping a bit...
2024-07-09 15:04:54,085 DEBUG [io.deb.con.bin.BinlogStreamingChangeEventSource] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Received query command: Event{header=EventHeaderV4{timestamp=1720517694000, eventType=QUERY, serverId=28029116, headerLength=19, dataLength=89, nextPosition=126073, flags=8}, data=QueryEventData{threadId=1184, executionTime=0, errorCode=0, database='test', sql='create database test'}}
2024-07-09 15:04:54,086 DEBUG [io.deb.con.bin.BinlogDatabaseSchema] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Processing streaming DDL 'create database test' for database 'test'
for DML
] (blc-company-db.cluster-cpw2c08moqyr.us-east-1.rds.amazonaws.com:3306) Received query command: Event{header=EventHeaderV4{timestamp=1720517320000, eventType=QUERY, serverId=28029116, headerLength=19, dataLength=59, nextPosition=123565, flags=8}, data=QueryEventData{threadId=1184, executionTime=0, errorCode=0, database='company', sql='BEGIN'}}Last I see this. It seems only ddl level are captured.
ismailsimsek
application.properties
logs:
MacBook-Pro:debezium-server-iceberg milan$ ./run.sh
CREATE DATABASE company;CREATE TABLE orders ( order_id INT AUTO_INCREMENT PRIMARY KEY, customer_id INT NOT NULL, order_date DATE NOT NULL, total_amount DECIMAL(10, 2) NOT NULL); INSERT INTO orders (customer_id, order_date, total_amount) VALUES (1, '2023-01-15', 150.50);INSERT INTO orders (customer_id, order_date, total_amount) VALUES (2, '2023-02-20', 200.00); INSERT INTO orders (customer_id, order_date, total_amount) VALUES (3, '2023-03-25', 320.75); INSERT INTO orders (customer_id, order_date, total_amount) VALUES (4, '2023-04-10', 450.00); INSERT INTO orders (customer_id, order_date, total_amount) VALUES (5, '2023-05-05', 500.25);
I don't see these events streaming to my s3. Can anyone help identify the error in config ?