search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.81k stars 1.76k forks source link

Failed to upsert for commit time 20240822175004430 #7462

Open mtzcsc opened 3 weeks ago

mtzcsc commented 3 weeks ago

Search before asking

What happened

MySQL reports an error when sending offline data to Hudi

SeaTunnel Version

2.3.7

SeaTunnel Config

env {
  parallelism = 1
  job.mode = "BATCH"
}

source{
    Jdbc {
        url = "jdbc:mysql://192.168.31.247:3306/test?serverTimezone=GMT%2b8&useUnicode=true&characterEncoding=UTF-8&rewriteBatchedStatements=true"
        driver = "com.mysql.cj.jdbc.Driver"
        connection_check_timeout_sec = 100
        user = "root"
        password = "digi"
        table_path = "test.user"
        partition_column = "id"
    }
}

transform {
}

sink {
  Hudi {
    table_dfs_path = "hdfs://192.168.144.131:9000/hadoop/hudi/"
    table_name = "user"
    table_type = "copy_on_write"
  }
}

Running Command

./bin/seatunnel.sh --config ./config/test.conf -e local

Error Exception

2024-08-22 17:50:05,255 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,

2024-08-22 17:50:05,255 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-08-22 17:50:05,255 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed

2024-08-22 17:50:05,256 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:211)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:292)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:74)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:70)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:717)
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1039)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:234)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:208)
        ... 16 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:232)
        ... 17 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64)
        at org.apache.hudi.table.action.commit.JavaInsertCommitActionExecutor.execute(JavaInsertCommitActionExecutor.java:46)
        at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:108)
        at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:84)
        at org.apache.hudi.client.HoodieJavaWriteClient.insert(HoodieJavaWriteClient.java:137)
        at org.apache.seatunnel.connectors.seatunnel.hudi.sink.writer.HudiSinkWriter.flush(HudiSinkWriter.java:316)
        at org.apache.seatunnel.connectors.seatunnel.hudi.sink.writer.HudiSinkWriter.prepareCommit(HudiSinkWriter.java:174)
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.lambda$prepareCommit$4(MultiTableSinkWriter.java:217)
        ... 5 more
Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.lambda$execute$0(BaseJavaCommitActionExecutor.java:123)
        at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:119)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:70)
        at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:58)
        ... 12 more
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:71)
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:37)
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
        ... 18 more
Caused by: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:75)
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:67)
        ... 20 more
Caused by: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:85)
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:94)
        at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:144)
        at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:437)
        at org.apache.hudi.io.hadoop.HoodieBaseParquetWriter.close(HoodieBaseParquetWriter.java:164)
        at org.apache.hudi.io.hadoop.HoodieAvroParquetWriter.close(HoodieAvroParquetWriter.java:88)
        at org.apache.hudi.io.HoodieCreateHandle.close(HoodieCreateHandle.java:216)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.closeOpenHandles(CopyOnWriteInsertHandler.java:126)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.finish(CopyOnWriteInsertHandler.java:119)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.finish(CopyOnWriteInsertHandler.java:44)
        at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:72)
        ... 21 more
Caused by: java.nio.channels.ClosedChannelException
        at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73)
        at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153)
        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:623)
        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:578)
        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134)
        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134)
        at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:65)
        at org.apache.parquet.hadoop.ParquetFileWriter.close(ParquetFileWriter.java:1663)
        at org.apache.parquet.util.AutoCloseables.close(AutoCloseables.java:49)
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:83)
        ... 31 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:203)
        ... 2 more

2024-08-22 17:50:05,257 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
===============================================================================

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:211)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:292)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:74)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:70)
        at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:50)
        at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:51)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:168)
        at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78)
        at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:717)
        at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1039)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:234)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:208)
        ... 16 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.prepareCommit(MultiTableSinkWriter.java:232)
        ... 17 more
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20240822175004430
        at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64)
        at org.apache.hudi.table.action.commit.JavaInsertCommitActionExecutor.execute(JavaInsertCommitActionExecutor.java:46)
        at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:108)
        at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:84)
        at org.apache.hudi.client.HoodieJavaWriteClient.insert(HoodieJavaWriteClient.java:137)
        at org.apache.seatunnel.connectors.seatunnel.hudi.sink.writer.HudiSinkWriter.flush(HudiSinkWriter.java:316)
        at org.apache.seatunnel.connectors.seatunnel.hudi.sink.writer.HudiSinkWriter.prepareCommit(HudiSinkWriter.java:174)
        at org.apache.seatunnel.api.sink.multitablesink.MultiTableSinkWriter.lambda$prepareCommit$4(MultiTableSinkWriter.java:217)
        ... 5 more
Caused by: java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.lambda$execute$0(BaseJavaCommitActionExecutor.java:123)
        at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:119)
        at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:70)
        at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:58)
        ... 12 more
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:71)
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:37)
        at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119)
        ... 18 more
Caused by: org.apache.hudi.exception.HoodieException: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:75)
        at org.apache.hudi.execution.JavaLazyInsertIterable.computeNext(JavaLazyInsertIterable.java:67)
        ... 20 more
Caused by: org.apache.parquet.util.AutoCloseables$ParquetCloseResourceException: Unable to close resource
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:85)
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:94)
        at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:144)
        at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:437)
        at org.apache.hudi.io.hadoop.HoodieBaseParquetWriter.close(HoodieBaseParquetWriter.java:164)
        at org.apache.hudi.io.hadoop.HoodieAvroParquetWriter.close(HoodieAvroParquetWriter.java:88)
        at org.apache.hudi.io.HoodieCreateHandle.close(HoodieCreateHandle.java:216)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.closeOpenHandles(CopyOnWriteInsertHandler.java:126)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.finish(CopyOnWriteInsertHandler.java:119)
        at org.apache.hudi.execution.CopyOnWriteInsertHandler.finish(CopyOnWriteInsertHandler.java:44)
        at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:72)
        ... 21 more
Caused by: java.nio.channels.ClosedChannelException
        at org.apache.hadoop.hdfs.ExceptionLastSeen.throwException4Close(ExceptionLastSeen.java:73)
        at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:153)
        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:623)
        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:578)
        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134)
        at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134)
        at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.close(HadoopPositionOutputStream.java:65)
        at org.apache.parquet.hadoop.ParquetFileWriter.close(ParquetFileWriter.java:1663)
        at org.apache.parquet.util.AutoCloseables.close(AutoCloseables.java:49)
        at org.apache.parquet.util.AutoCloseables.uncheckedClose(AutoCloseables.java:83)
        ... 31 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:203)
        ... 2 more
2024-08-22 17:50:05,259 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-1] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

image

Are you willing to submit PR?

Code of Conduct

Hisoka-X commented 3 weeks ago

cc @liugddx