Closed Jetiaime closed 5 months ago
It shows the shaded hudi google guava jar is missing.
It shows the shaded hudi google guava jar is missing.
Yes, And I compiled again without -Pspark-bundle-shade-hive
, and it worked fine so far. I'm comfused by this profile, why it should be added in flink like-Pflink-bundle-shade-hive3
, but not in spark ?
flink use it for avoiding the jar conflicts with other Hive related jars, not sure whether Spark needs this.
@Jetiaime Spark dont need it.
Describe the problem you faced
I'm trying to compile Hudi 0.12.2 with Spark 3.3.1, Hadoop 3.3.2, Hive 3.1.2. And I compiled HBase 2.4.9 With Hadoop3 in order to avoid HUDI-4805, And I excluded a lot of jetty dependencies to avoid
java.lang.NoSuchMethodError: org.apache.hudi.org.apache.jetty.server.session.SessionHandler.setHttpOnly(Z)V
. After doing this, I compiled withmvn clean package -Pscala-2.12 -Pspark3.3 -Pspark-bundle-shade-hive -DskipTests -Dcheckstyle.skip
On my macbook m1pro chips.And I got this:
Here's My Dependencies that I exclude:
Environment Description
Hudi version : 0.12.2
Spark version : 3.3.1
Hive version : 3.1.2
Hadoop version : 3.3.2
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) : no
Additional context
I'm using SparkSQL to insert into a very simple Hudi table:
Stacktrace