sparkxgb : Xgboost in Databricks

rstudio / sparkxgb

R interface for XGBoost on Spark

https://spark.posit.co/packages/sparkxgb/

Other

47 stars 14 forks source link

sparkxgb : Xgboost in Databricks #41

Open Beginerxyz opened 3 years ago

Beginerxyz commented 3 years ago

Error : java.lang.ClassNotFoundException: ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier Error : java.lang.ClassNotFoundException: ml.dmlc.xgboost4j.scala.spark.XGBoostClassifier.

Does Databricks not support using extensions - sparkxgb to run xgboost model ?

thomasjohnflaherty commented 2 years ago

Doesn't look like this runs on high concurrency clusters. I ran into the same issue as the op but everything ran fine on standard.

Edit: looks like I'm getting that error again. Switched to runtime 10.0 ML with the following Spark config:

spark.sql.execution.arrow.sparkr.enabled true spark.databricks.delta.preview.enabled true