search

awesome-kyuubi / hadoop-testing

Testing Sandbox for Hadoop Ecosystem Components
Apache License 2.0
32 stars 13 forks source link

Hudi bundle jar of spark 3.5 cannot be found #100

Open wForget opened 6 months ago

wForget commented 6 months ago

download error:

downloading hudi-spark3.5-bundle_2.12-0.14.1.jar ...
--2024-05-17 13:58:30--  https://github.com/yanghua/incubator-hudi/releases/download/release-0.14.1/hudi-spark3.5-bundle_2.12-0.14.1.jar
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-05-17 13:58:31 ERROR 404: Not Found.

cc @yanghua

pan3793 commented 5 months ago

Hudi 0.15.0 is released with pre-built bundle jars that support Spark 3.5 and Scala 2.13, not sure about Hadoop 3

yanghua commented 5 months ago

download error:

downloading hudi-spark3.5-bundle_2.12-0.14.1.jar ...
--2024-05-17 13:58:30--  https://github.com/yanghua/incubator-hudi/releases/download/release-0.14.1/hudi-spark3.5-bundle_2.12-0.14.1.jar
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-05-17 13:58:31 ERROR 404: Not Found.

cc @yanghua

It seems that you changed the spark version to 3.5. While I only provided 3.4. This personal binary package was tried to resolve the conflict from the dependency of hbase. Not sure if the issue has been resolved in hudi 0.15.

pan3793 commented 5 months ago

@yanghua not sure about Hudi 0.15, but Spark 3.5 drops support for Hadoop2

yanghua commented 5 months ago

@yanghua not sure about Hudi 0.15, but Spark 3.5 drops support for Hadoop2

Got it, will try to verify if the official hudi bundle for spark 3.5 works fine.