kazdy
commented
1 year ago I inspected all files that were source files for clustering and all have the same schema. The file created as result of clustering has changed schema, column named "year" was moved from the middle of the file schema to the end of the schema.
before clustering: awesome_column_1 awesome_column_1 year month header
after clustering awesome_column_1 awesome_column_1 month header year
No other column was relocated.
It looks like during clustering schema was modified and now it's causing issues.
Interestingly I have tables with exactly the same schema where clustering did not break anything after enabling async clustering. The only difference is that here year column is used as the only partition column, in the other table where everything is fine I'm using year and month as partition columns..
UPDATE: ^^ that's no longer true, table where year and month was used as parition columns is also broken after clustering and I can not write to it anymore. Encountering the same errors. This time 3 files were clustered one from partition year=2023/month=02 and one from year=2023/month=03.
In this case files from year=2022 and year=2023 were clustered and two files (one for 2022 and one for 2023) were created.
When it comes to "fixing" this table I think I can just rollback to the commit before clustering (last before replacecommit) and it should be fine? And I guess it's a good idea to disable clustering just to be safe.
Similar to #5262 where user also claims no explicit changes to record schema.
danny0405
mansipp
xushiyan
VIKASPATID
VitoMakarevich
Describe the problem you faced
I'm observing failed spark stages
"Doing partition and writing data: <table name>" in " SparkUpsertCommitActionExecutor" job group.Effectively it can no longer insert or update records. There were no changes to schema or datatypes, I only switched from inline to async services for CoW table that is updated by spark structured streaming job. Only one job suffers from this issue after switching to async services. I can see that async services are running fine eg cleaning. I'm using in-process lock, metadata table is enabled.After reviewing the file itself
I don't see anything obviously wrong with it when it comes to columns and types.I see that one column was relocated from the middle of the schema to the end of the schema in the parquet file itself. The file that's causing issues was created by clustering action. I checked it by looking at commits, the only commit related to this file is a replacecommit.EDIT (22/03/2023): Note that "schema" in replacecommit file has "year" column in the middle of the schema, for some reason it's only relocated in the file created by clustering.
To Reproduce
Steps to reproduce the behavior:
run async clustering, do not change record schema, partition column is relocated to the end of the schema
EDIT (23/03/2023): Fully reproduced example with hudi files
https://github.com/kazdy/HUDI-8259/tree/mainEDIT (30/03/2023) Updated repo with better example to reproduce (in replicated branch): https://github.com/kazdy/HUDI-8259Expected behavior
Hudi continues to write updates with no failed stages after clustering is done.
Environment Description
Hudi version : 0.12.1-amzn
Spark version : 3.3.0-amzn-1
Hive version : na
Hadoop version : na
Storage (HDFS/S3/GCS..) : s3
Running on Docker? (yes/no) : no, running in emr serverless
Additional context
Previous job run was stopped (killed) while Hudi was writing data, in this job that we see the issue first action taken was rollback. It can be that something went wrong with this rollback? I also see that async clustering was running, maybe file was corrupted while being rewritten?
Stacktrace