search

awesome-kyuubi / hadoop-testing

Testing Sandbox for Hadoop Ecosystem Components
Apache License 2.0
29 stars 11 forks source link

Hudi bundle jar of spark 3.5 cannot be found #100

Open wForget opened 1 month ago

wForget commented 1 month ago

download error:

downloading hudi-spark3.5-bundle_2.12-0.14.1.jar ...
--2024-05-17 13:58:30--  https://github.com/yanghua/incubator-hudi/releases/download/release-0.14.1/hudi-spark3.5-bundle_2.12-0.14.1.jar
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-05-17 13:58:31 ERROR 404: Not Found.

cc @yanghua

pan3793 commented 4 weeks ago

Hudi 0.15.0 is released with pre-built bundle jars that support Spark 3.5 and Scala 2.13, not sure about Hadoop 3

yanghua commented 4 weeks ago

download error:

downloading hudi-spark3.5-bundle_2.12-0.14.1.jar ...
--2024-05-17 13:58:30--  https://github.com/yanghua/incubator-hudi/releases/download/release-0.14.1/hudi-spark3.5-bundle_2.12-0.14.1.jar
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-05-17 13:58:31 ERROR 404: Not Found.

cc @yanghua

It seems that you changed the spark version to 3.5. While I only provided 3.4. This personal binary package was tried to resolve the conflict from the dependency of hbase. Not sure if the issue has been resolved in hudi 0.15.

pan3793 commented 4 weeks ago

@yanghua not sure about Hudi 0.15, but Spark 3.5 drops support for Hadoop2

yanghua commented 3 weeks ago

@yanghua not sure about Hudi 0.15, but Spark 3.5 drops support for Hadoop2

Got it, will try to verify if the official hudi bundle for spark 3.5 works fine.