Closed XuQianJin-Stars closed 1 year ago
@XuQianJin-Stars Can you elaborate more on the setup and environment and steps to reproduce?
also met this error today. Caused by: org.apache.hudi.exception.HoodieIOException: Could not read commit details from hdfs://xxx/.hoodie/20220720100959953.compaction.requested at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.readDataFromPath(HoodieActiveTimeline.java:624) at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.readCompactionPlanAsBytes(HoodieActiveTimeline.java:266) at org.apache.hudi.common.util.CompactionUtils.getCompactionPlan(CompactionUtils.java:148) at org.apache.hudi.sink.compact.CompactionCommitSink.lambda$commitIfNecessary$1(CompactionCommitSink.java:116) at java.util.HashMap.computeIfAbsent(HashMap.java:1127) at org.apache.hudi.sink.compact.CompactionCommitSink.commitIfNecessary(CompactionCommitSink.java:114) at org.apache.hudi.sink.compact.CompactionCommitSink.invoke(CompactionCommitSink.java:103) at org.apache.hudi.sink.compact.CompactionCommitSink.invoke(CompactionCommitSink.java:54) at org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:54) at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) at org.apache.flink.streaming.runtime.tasks.S
@XuQianJin-Stars : gentle reminder to respond to above's request/questions.
@XuQianJin-Stars @fengjian428 : did you folks inspect the .hoodie folder. by any chance the delta commit or commit file of interest seen in stacktrace is 0 byte file? does the error goes away if you restart the pipeline? or its just stuck. if you can share the contents of .hoodie, we can dig in to see whats happening.
closed as discussed offline, this issue is not re-occurring now.
This happened for us and the root cause is we have concurrent running spark jobs writing to the same insert_overwrite table, different partition. Our jobs didn't have proper concurrent locking configurations. After adding DynamoDB Locks for these insert_overwrite write tasks, we stopped seeing these issues anymore.