Closed VitoMakarevich closed 5 months ago
I went deeper and found that in Hudi 0.13.x and 0.14.x clustering do not honor flag hoodie.datasource.write.drop.partition.columns=true,
Is this a bug?
It looks to be, and in this version(0.12.x) it causes Upsert unresolvable exception
Does a higher release 0.14.x solve the problem already? Did you have a chance to try that?
0.13.x solves this, I mean it's writing files with a partition column but it does not cause an exception for upsert
path. Unfortunately we cannot upgrade, so I'm asking for a workaround way:
because it's written into hoodie.properties file - if I change the contents of this file
yeah, it's a table configuration instead of write config. So the impact is global for read and write. cc @nsivabalan any ideas?
In the meantime, it looks like I found a workaround
"hoodie.datasource.write.row.writer.enable": "false"
for the time of clustering does the trick - once it's done, it likely can be switched back(will check).
This setting makes writing a bit slower but files are produced without partitionpath
column as per the initial configuration.
I also checked on some test tables that manually changing property in hoodie.properties
works, but then you must ensure that clustering touches all files - which may be problematic if you have a lot of partitions. Since it creates Npartitions * (clustering group per partition) - so in our case, it will be submitting ~5k spark jobs which 99% will blow up the driver. And there is no yet way to limit this number if you want to run clustering on all partitions(I see in newer Hudi versions there is executor service which handles this).
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
Hudi 0.12.2 version. Hello, we are now using Hudi COW with
hoodie.datasource.write.drop.partition.columns=true
(non-default). And recently we wanted to run clustering for this dataset. The problem we observe is thatbulk_insert
used in clustering does not honor this setting andparquet
files are created with partition columns. As far as I checked it's not a concern for reading via Spark, but it causes problem for upserting for those partitions where we had clustering. If an update comes to such file, in 0.12.x we see failures likeParquet/Avro schema mismatch: Avro field 'partition1' not found
. Reproduction here I went deeper and found that in Hudi 0.13.x and 0.14.x clustering do not honor flaghoodie.datasource.write.drop.partition.columns=true
, but there it does not cause problems with upsert - partitions can udpated. Given this, we probably cannot upgrade now, but would like to execute clustering. So I need help with the following question:hoodie.datasource.write.drop.partition.columns
fromtrue
tofalse
? As far as I observed it looks to be working, but.hoodie.properties
file - if I change the contents of this file - it works this way: subsequent update appends partition column value, but only for changed rows, old remains withnull
. However, in the read path, old rows have this field populated(some magic). And once clustering is run - all rows have field populated. Important: we would not be able to execute clustering all table at once - likely, so if we run it in batches, is it safe to have part of the files including partition columns and part not?Nice to answer questions:
To Reproduce
Reproduction here
Environment Description
Hudi version : 0.12.2 - 0.14.x
Spark version : 3.3.0
Additional context
Add any other context about the problem here.
Stacktrace