[SUPPORT] using spark's observe feature on dataframes saved by hudi is stuck

[szingerpeter]

Describe the problem you faced

When trying to use the observe function on dataframes saved by hudi the application gets stuck after saving the data and trying to retrieve the statistics.

To Reproduce

Steps to reproduce the behavior:

from pyspark.sql import DataFrame, Observation
from pyspark.sql import functions as F
observation = Observation()

df = spark.createDataFrame([[1, 1], [2, 2], [3, 3], [4, 4]])

df = df.observe(observation, F.count(F.lit(1)).alias('row_count'))

df.write.format('csv').mode('overwrite').save('file:/opt/spark/work-dir/test_csv')

observation.get # returns: {'row_count': 4}

observation2 = Observation()
df2 = spark.createDataFrame([[1, 1], [2, 2], [3, 3], [4, 4]])

hudi_options = {
    'hoodie.table.name': 'test',
    'hoodie.datasource.write.recordkey.field': '_1',
    'hoodie.datasource.write.partitionpath.field': '',
    'hoodie.datasource.write.table.name': 'test',
    'hoodie.datasource.write.operation': 'insert_overwrite',
    'hoodie.datasource.write.precombine.field': '_2',
}

df2 = df2.observe(observation2, F.count(F.lit(1)).alias('row_count'))

df.write.format("hudi").\
    options(**hudi_options).\
    mode("overwrite").\
    save('file:/opt/spark/work-dir/test')

observation2.get # gets stuck

Disclaimer: I know there are hudi metrics and callbacks; however, i would like to add some more advanced quality checks to our applications

Environment Description

• Hudi version : 0.13.0-amzn-0

• Spark version : 3.3.2

• Hive version : 3.1.3

• Hadoop version : 3.3.3

• Storage (HDFS/S3/GCS..) : HDFS/S3 both

• Running on Docker? (yes/no) : both

[szingerpeter]

@codope is there anything else needed from my side at the moment?

[ad1happy2go]

@szingerpeter I will look into it. Sorry for the delay here.

[szingerpeter]

@ad1happy2go , thank you!

[szingerpeter]

@ad1happy2go , did you have a chance to take a look at the issue?

[ad1happy2go]

@szingerpeter Sorry again. I got swamped in some other urgent tasks. Will try to look into it by end of this week.