search

memiiso / debezium-server-iceberg

Replicates any database (CDC events) to Apache Iceberg (To Cloud Storage)
Apache License 2.0
184 stars 35 forks source link

Cannot invoke "Object.equals(Object)" because the return value of "org.apache.iceberg.data.Record.getField(String)" is null #365

Open GOVINDARAMTEKKAR97 opened 3 months ago

GOVINDARAMTEKKAR97 commented 3 months ago

applicaton-properties.txt Hi @ismailsimsek , hope you are in a great health. I am trying to do cdc for oracle database . I am continuously facing issue that Mentioned here the error datails are below Exception: Cannot invoke "Object.equals(Object)" because the return value of "org.apache.iceberg.data.Record.getField(String)" is null at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at io.debezium.embedded.async.AsyncEmbeddedEngine.runTasksPolling(AsyncEmbeddedEngine.java:468) at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:212) at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:178) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.lang.NullPointerException: Cannot invoke "Object.equals(Object)" because the return value of "org.apache.iceberg.data.Record.getField(String)" is null at io.debezium.server.iceberg.tableoperator.BaseDeltaTaskWriter.write(BaseDeltaTaskWriter.java:53) at io.debezium.server.iceberg.tableoperator.BaseDeltaTaskWriter.write(BaseDeltaTaskWriter.java:16) at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTablePerSchema(IcebergTableOperator.java:173) at io.debezium.server.iceberg.tableoperator.IcebergTableOperator.addToTable(IcebergTableOperator.java:157) at io.debezium.server.iceberg.IcebergChangeConsumer.handleBatch(IcebergChangeConsumer.java:165) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:56) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1157) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1138) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ... 3 more Also I am not getting rows and columns and tables as I am single oracle database with one tables. I will be adding applications.properties

Please let us know what need to changes in application.properties. It will help me a lot. Regards, Govinda

ismailsimsek commented 3 months ago

@GOVINDARAMTEKKAR97
this is related upsert mode https://github.com/memiiso/debezium-server-iceberg/blob/a96d4fc14310ec0579a1d4b84605d3e180017326/debezium-server-iceberg-sink/src/main/java/io/debezium/server/iceberg/tableoperator/BaseDeltaTaskWriter.java#L54

this means you are receiving events with null __op field. could you check the events and see why its generated without this field?

GOVINDARAMTEKKAR97 commented 3 months ago

Hi @ismailsimsek thanks for suggestion, after successfully executing I am facing another error that is failed to create table with prefix and shema name I have added the below error details

2024-06-18 16:40:05,467 ERROR [io.deb.emb.asy.AsyncEmbeddedEngine] (pool-7-thread-1) Engine has failed with : java.util.concurrent.ExecutionException: io.debezium.DebeziumException: Failed to create table from debezium event schema:default.debeziumcdc_tutorial_CDHRUV_TEST Error:Invalid table identifier: default.debeziumcdc_tutorial_C__DHRUV_TEST at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at io.debezium.embedded.async.AsyncEmbeddedEngine.runTasksPolling(AsyncEmbeddedEngine.java:468) at io.debezium.embedded.async.AsyncEmbeddedEngine.run(AsyncEmbeddedEngine.java:212) at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:178) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.debezium.DebeziumException: Failed to create table from debezium event schema:default.debeziumcdc_tutorial_CDHRUV_TEST Error:Invalid table identifier: default.debeziumcdc_tutorial_C__DHRUV_TEST at io.debezium.server.iceberg.IcebergChangeConsumer.lambda$loadIcebergTable$1(IcebergChangeConsumer.java:194) at java.base/java.util.Optional.orElseGet(Optional.java:364) at io.debezium.server.iceberg.IcebergChangeConsumer.loadIcebergTable(IcebergChangeConsumer.java:187) at io.debezium.server.iceberg.IcebergChangeConsumer.handleBatch(IcebergChangeConsumer.java:164) at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:56) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1157) at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1138) at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ... 3 more Caused by: java.lang.IllegalArgumentException: Invalid table identifier: default.debeziumcdc_tutorial_C__DHRUV_TEST at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:218) at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.(BaseMetastoreCatalog.java:148) at org.apache.iceberg.BaseMetastoreCatalog.buildTable(BaseMetastoreCatalog.java:96) at io.debezium.server.iceberg.IcebergUtil.createIcebergTable(IcebergUtil.java:92) at io.debezium.server.iceberg.IcebergChangeConsumer.lambda$loadIcebergTable$1(IcebergChangeConsumer.java:192) ... 13 more kindly suggest what permission issue beacuse we have given all priviledges that mentioned in offical document of oracle. Regards, Govinda

ismailsimsek commented 3 months ago

please make sure you are using correct namespace and table naming which is compatible with the destination catalog, glue catalog.

GOVINDARAMTEKKAR97 commented 3 months ago

image i am getting error while creating table by iceberg but same naming convention have used in postgress work image

ismailsimsek commented 3 months ago

@GOVINDARAMTEKKAR97 you could try to create same table identifier in glue manually and see what happens

Invalid table identifier: default.debeziumcdc_tutorial_C__DHRUV_TEST

GOVINDARAMTEKKAR97 commented 3 months ago

Hi @ismailsimsek , I have tried this thing earlier but still facing the error, would please suggest us any configuration or if you can create any document for oracle debezium iceberg procedure, beacasue we are facing this too much error while setup configuration, we are working since more than month on this thing. Please revert back us for solutions

Regards, Govinda