search

sllynn / spark-xgboost

A Python wrapper for XGBoost4J-Spark classes.
https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html
Other
47 stars 13 forks source link

java.lang.NoClassDefFoundError : org/spache/spark/ml/param/shared/HasWeightCol$class #5

Closed maheshv26 closed 3 years ago

maheshv26 commented 3 years ago

On databricks envrionment, getting the java.lang.NoClassDefFound error when running the XGBoost Classifier function. Spark version is 3.01. Can you please help?

sllynn commented 3 years ago

Which runtime are you using? How are you attaching the library?

maheshv26 commented 3 years ago

I am using DBR 7.3 LTS. xgboost-linux64 version: 0.8.3-spark2.3-s_2.11 has been installed on the cluster

sllynn commented 3 years ago

The xgboost version you're using is built for scala 2.11 & spark 2.3 so is unlikely to work with this wrapper or any 7.x runtime.

Can you try again with DBR 7.3 ML?

You won't need to install the xgboost library as it's pre-packaged with the runtime. See the release notes here: https://docs.databricks.com/release-notes/runtime/7.3ml.html#java-and-scala-libraries-scala-212-cluster