search

torvalds-dev / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
0 stars 0 forks source link

Compaction/Clustering job not fail when throw HoodieException #26

Open 0xDTE opened 9 months ago

0xDTE commented 9 months ago

Clustering/Compaction job throw follow exception, the final result returns -1 and the job's state is success.

ERROR UtilHelpers: Cluster failed org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 2 times, most recent failure: Lost task 0.1 in stage 4.0 (TID 5) : org.apache.hudi.exception.HoodieException: unable to read next record from parquet file at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:54) at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39) at org.apache.hudi.common.util.collection.MappingIterator.hasNext(MappingIterator.java:39) at org.apache.hudi.client.utils.ConcatenatingIterator.hasNext(ConcatenatingIterator.java:45) Issue Links links to Web Link GitHub Pull Request #10050
Activity All Comments Work Log History Activity Transitions

danny0405 Danny Chen added a comment - 02/Dec/23 03:43 Fixed via master branch: 77e2f42c6e49478cadf2b6bc5c732ee8673a8d69

People Assignee: Unassigned Unassigned Reporter: ksmou kwang Votes: 0 Vote for this issue Watchers: 2 Start watching this issue Dates Created: 10/Nov/23 08:36 Updated: Yesterday 03:43

torvalds-dev commented 9 months ago

torvalds.dev is analyzing the ticket

torvalds-dev commented 9 months ago

The issue you're referring to is related to Apache Hudi, a data lake framework that provides stream processing and incremental data pipeline capabilities. The specific problem is with the compaction and clustering jobs in Hudi.

From the error message, it seems that the compaction/clustering job is failing due to an exception (HoodieException) being thrown. This exception is caused by an inability to read the next record from a Parquet file. Parquet is a columnar storage file format that is optimized for use with Apache Hadoop due to its superior query performance and lower storage cost.

Despite this failure, the job's state is reported as successful, and the final result returns -1. This is likely a bug, as a job that encounters a failure should not report a successful state.

The issue was reported by a user named ksmou and was not assigned to anyone. It was created on 10/Nov/23 and last updated yesterday.

A user named Danny Chen commented that the issue was fixed via the master branch with the commit 77e2f42c6e49478cadf2b6bc5c732ee8673a8d69. You can view the changes made in this commit by visiting the provided GitHub link.

As for the code you provided, it seems to be a test case for the Hudi framework. It's creating a Parquet table, simulating inserts, setting up incremental reads, and verifying that incremental reads do not show inserts after a compaction timestamp. It also tests the conversion of Avro records to ArrayWritable objects.

If you have any specific questions about the code or the issue, feel free to ask!