ad1happy2go
commented
12 months ago @fearlsgroove Can you share the table and writer configurations.
Open fearlsgroove opened 12 months ago
ad1happy2go
commented
12 months ago @fearlsgroove Can you share the table and writer configurations.
fearlsgroove
commented
12 months ago Sure thanks for the reply:
"hoodie.datasource.write.recordkey.field": "id,item_date",
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.write.partitionpath.field": "partition_year",
"hoodie.datasource.write.table.name": "the_table_name",
"hoodie.datasource.write.operation": "upsert",
"hoodie.datasource.write.precombine.field": "event_date",
"hoodie.datasource.write.hive_style_partitioning": True,
"hoodie.datasource.write.table.type": "MERGE_ON_READ",
"hoodie.clustering.execution.strategy.class": "org.apache.hudi.client.clustering.run.strategy.SparkSortAndSizeExecutionStrategy",
"hoodie.clustering.plan.strategy.sort.columns": "id",
"hoodie.upsert.shuffle.parallelism": 32,
"hoodie.insert.shuffle.parallelism": 32,
"hoodie.bulkinsert.shuffle.parallelism": 32,
"hoodie.clean.async": True,
"hoodie.clustering.async.enabled": True,
"hoodie.clustering.async.max.commits": 8, # default is 4
# Seems to avoid some concurrency/retry errors. Possibly ineffective
"hoodie.clustering.updates.strategy": "org.apache.hudi.client.clustering.update.strategy.SparkAllowUpdateStrategy",@fearlsgroove By stack trace, it looks like you got some data which is of higher precision i.e 13 for that column. Do you have any way to confirm if that is not the case. Although not sure why it didn't failed while writing log files.
I will try to insert the similar data to confirm if it fails in writer or compaction.
fearlsgroove
commented
12 months ago Yea I was thinking that as well but I don't see how it could be the case. We did evolve the schema in the source topic recently, but the only decimal field in the data did not change, and I'm 99% sure we've only pulled records with the new schema anyway.
If I look at all the data i can pull from the topic i don't see any fields with decimal precision that would indicate the problem. Is there any way you can suggest to inspect the records in logs that are going to be compacted so I can try to confirm and/or correct it?
ad1happy2go
commented
12 months ago @fearlsgroove Can you try to read the data (without compaction) and see if you are able to and if data looks correct?
fearlsgroove
commented
12 months ago Yeap i can read the data succcessfully. Schema shows the only decimal value as a (12,2) value which is what I expect. Is there a way to try to read whatever records in the logs due for compaction might be causing this issue?
ad1happy2go
commented
11 months ago Can you try setting spark.sql.storeAssignmentPolicy=legacy
Referring this issue - https://github.com/apache/hudi/issues/6209
ad1happy2go
commented
11 months ago @fearlsgroove Were you able to resolve it. Let us know if you still face this issue. Thanks.
watermelon12138
commented
11 months ago @fearlsgroove we will meet this problem only if oldSchema and writeSchema have diff. The writeSchema comes from the input data. So, I suggest you to check the schema in deltacommit metadata file. These meta files generally saved in /xxx_hudi_table/.hoodie/ directory.
nsivabalan
commented
6 months ago hey @ad1happy2go : reminder to follow up on this
I'm using spark structured streaming to ingest kafka records in avro format into a MoR hudi table. Somehow I've gotten something fouled up and compactions cannot complete. I'm not configuring any non-default settings for compaction, and spark tasks on the 'hoodiecompact' pool always fail with the stack below.
I do have a decimal field with precision 12 in the messages, but it hasn't changed to 13 anywhere. I don't know if it's possible to inspect the messages somehow to see if there are unexpectedly but I don't see how there could be.
Expected behavior The compaction completes without errors.
Environment Description
Stacktrace