liujinhui1994
commented
3 years ago 
Closed liujinhui1994 closed 3 years ago
liujinhui1994
commented
3 years ago
liujinhui1994
commented
3 years ago 
liujinhui1994
commented
3 years ago liujinhui1994
commented
3 years ago @n3nash @nsivabalan Can you help me? This problem has troubled me for a week. I tried my best to find the problem, but it still didn't solve it. Thank you very much~
pengzhiwei2018
commented
3 years ago java.lang.ClassNotFoundException: org.apache.hudi.DefaultSource at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
It seems that you have not put the hudi-spark bundle to the $SPARK_HOME/jars
liujinhui1994
commented
3 years ago No, please take a look at my description. The first batch of the same program can write data normally, but the second batch cannot be written. If it is a jar problem, the problem should also occur the first time.
------------------ Original ------------------ From: pengzhiwei @.> Date: Thu,Jul 15,2021 10:33 PM To: apache/hudi @.> Cc: liujinhui @.>, Author @.> Subject: Re: [apache/hudi] [SUPPORT] Use structedstreaming to consume kafka to write to hudi error (#3280)
liujinhui1994
commented
3 years ago Is there a partner who can help me?
nsivabalan
commented
3 years ago I could see two exceptions from your stacktrace:
@liujinhui1994 :
@vinothchandar @n3nash : Do you folks know of any changes that went in after 0.8.0 that might be of interest to us.
liujinhui1994
commented
3 years ago liujinhui1994
commented
3 years ago The latest discovery is that when I turn off hive sync I don't get the above error, when I turn it on I get the above error @nsivabalan
vinothchandar
commented
3 years ago @codope can we do a git blame of the spark bundle to see what has changed since?
codope
commented
3 years ago @liujinhui1994 I tried to reproduce this locally but for me the streaming ran fine. My setup is Spark 2.4.7, Hadoop 2.7.3 and Hive 2.3.7. I tried for both COW and MOR table
val query = df.writeStream.queryName("demo").foreachBatch { (batchDF: DataFrame, _: Long) => {
batchDF.persist()
batchDF.write.format("org.apache.hudi")
.option(TABLE_TYPE_OPT_KEY.key, "COPY_ON_WRITE")
.option(PRECOMBINE_FIELD_OPT_KEY.key, "kafka_timestamp")
.option(RECORDKEY_FIELD_OPT_KEY.key, "kafka_partition_offset")
.option(PARTITIONPATH_FIELD_OPT_KEY.key, "partition_date")
.option(TABLE_NAME_OPT_KEY.key, "copy_on_write_table")
.option("hoodie.table.name", "copy_on_write_table")
.option(HIVE_SYNC_ENABLED_OPT_KEY.key, true)
.option(HIVE_STYLE_PARTITIONING_OPT_KEY.key, true)
.mode(SaveMode.Append)
.save("/tmp/sparkHudi/COPY_ON_WRITE")
batchDF.write.format("org.apache.hudi")
.option(TABLE_TYPE_OPT_KEY.key, "MERGE_ON_READ")
.option(TABLE_TYPE_OPT_KEY.key, "COPY_ON_WRITE")
.option(PRECOMBINE_FIELD_OPT_KEY.key, "kafka_timestamp")
.option(RECORDKEY_FIELD_OPT_KEY.key, "kafka_partition_offset")
.option(PARTITIONPATH_FIELD_OPT_KEY.key, "partition_date")
.option(TABLE_NAME_OPT_KEY.key, "merge_on_read_table")
.option("hoodie.table.name", "merge_on_read_table")
.option(HIVE_SYNC_ENABLED_OPT_KEY.key, true)
.option(HIVE_STYLE_PARTITIONING_OPT_KEY.key, true)
.mode(SaveMode.Append)
.save("/tmp/sparkHudi/MERGE_ON_READ")
batchDF.unpersist()
}}.option("checkpointLocation", "/tmp/sparkHudi/checkpoint/").start()Could you try again with the latest master?
liujinhui1994
commented
3 years ago I re-synchronized the latest code this time, and now it works. I think a lot of content has been updated recently. It should have been repaired.I have informed my colleagues that they are now also updating to the latest version. I found it was also successful @codope
nsivabalan
commented
3 years ago thanks @codope
liujinhui1994
commented
3 years ago @nsivabalan @codope @vinothchandar thanks
Tips before filing an issue
Have you gone through our FAQs?
Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
If you have triaged this as a bug, then file an issue directly.
Describe the problem you faced
A clear and concise description of the problem.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version : latest master
Spark version : 2.4.5
Hive version : not involving
Hadoop version : 3.1.1
Storage (HDFS/S3/GCS..) : obs
Running on Docker? (yes/no) : no
Additional context
Use structedstreaming to consume kafka to write to hudi. When using other versions of hudi, such as 0.7, 0.8, the program can run continuously and stably.
I tried to locate the problem, but did not find the problem
Stacktrace
Add the stacktrace of the error.