danny0405
commented
5 months ago I went deeper and found that in Hudi 0.13.x and 0.14.x clustering do not honor flag hoodie.datasource.write.drop.partition.columns=true,
Is this a bug?
Closed VitoMakarevich closed 5 months ago
danny0405
commented
5 months ago I went deeper and found that in Hudi 0.13.x and 0.14.x clustering do not honor flag hoodie.datasource.write.drop.partition.columns=true,
Is this a bug?
VitoMakarevich
commented
5 months ago It looks to be, and in this version(0.12.x) it causes Upsert unresolvable exception
danny0405
commented
5 months ago Does a higher release 0.14.x solve the problem already? Did you have a chance to try that?
VitoMakarevich
commented
5 months ago 0.13.x solves this, I mean it's writing files with a partition column but it does not cause an exception for upsert path. Unfortunately we cannot upgrade, so I'm asking for a workaround way:
danny0405
commented
5 months ago because it's written into hoodie.properties file - if I change the contents of this file
yeah, it's a table configuration instead of write config. So the impact is global for read and write. cc @nsivabalan any ideas?
VitoMakarevich
commented
5 months ago In the meantime, it looks like I found a workaround
"hoodie.datasource.write.row.writer.enable": "false"
for the time of clustering does the trick - once it's done, it likely can be switched back(will check).
This setting makes writing a bit slower but files are produced without partitionpath column as per the initial configuration.
VitoMakarevich
commented
5 months ago I also checked on some test tables that manually changing property in hoodie.properties works, but then you must ensure that clustering touches all files - which may be problematic if you have a lot of partitions. Since it creates Npartitions * (clustering group per partition) - so in our case, it will be submitting ~5k spark jobs which 99% will blow up the driver. And there is no yet way to limit this number if you want to run clustering on all partitions(I see in newer Hudi versions there is executor service which handles this).
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
Hudi 0.12.2 version. Hello, we are now using Hudi COW with
hoodie.datasource.write.drop.partition.columns=true(non-default). And recently we wanted to run clustering for this dataset. The problem we observe is thatbulk_insertused in clustering does not honor this setting andparquetfiles are created with partition columns. As far as I checked it's not a concern for reading via Spark, but it causes problem for upserting for those partitions where we had clustering. If an update comes to such file, in 0.12.x we see failures likeParquet/Avro schema mismatch: Avro field 'partition1' not found. Reproduction here I went deeper and found that in Hudi 0.13.x and 0.14.x clustering do not honor flaghoodie.datasource.write.drop.partition.columns=true, but there it does not cause problems with upsert - partitions can udpated. Given this, we probably cannot upgrade now, but would like to execute clustering. So I need help with the following question:hoodie.datasource.write.drop.partition.columnsfromtruetofalse? As far as I observed it looks to be working, but.hoodie.propertiesfile - if I change the contents of this file - it works this way: subsequent update appends partition column value, but only for changed rows, old remains withnull. However, in the read path, old rows have this field populated(some magic). And once clustering is run - all rows have field populated. Important: we would not be able to execute clustering all table at once - likely, so if we run it in batches, is it safe to have part of the files including partition columns and part not?Nice to answer questions:
To Reproduce
Reproduction here
Environment Description
Hudi version : 0.12.2 - 0.14.x
Spark version : 3.3.0
Additional context
Add any other context about the problem here.
Stacktrace