search

combust / mleap

MLeap: Deploy ML Pipelines to Production
https://combust.github.io/mleap-docs/
Apache License 2.0
1.5k stars 310 forks source link

Exception in thread "Thread-4" java.lang.NoClassDefFoundError: ml/combust/bundle/HasBundleRegistry #845

Closed drei34 closed 1 year ago

drei34 commented 1 year ago

I'm not sure if this issue has a resolution. Why would this error happen and how can you fix it?

I'm on Java 8, mleap 0.20.0, Scala 2.12 and pyspark 3.1.3.

https://github.com/combust/mleap/issues/442

jsleight commented 1 year ago

Might be related to the issue in #842

edit: oh definitely. You just made another issue for it, I got you 😄

drei34 commented 1 year ago

Ha right it is related to #842 I guess but more related to #442. #442 was closed, but I am unsure why because the answer is not there. So I figure I can directly address that here ...

drei34 commented 1 year ago

This was resolved after we installed Java 11 and used the malest mleap. We used mleap-databricks-runtime-fat-assembly-0.22.0. You also need to change JAVA_HOME inside of spark. The command below might work.

`

We need this new Java since the new Mleap uses Java 11.

sudo apt-get -y install openjdk-11-jdk export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 pip install mleap

If you don't do this, Spark will complain about Python 3.7 and 3.8.

export PYSPARK_PYTHON=/opt/conda/miniconda3/bin/python export PYSPARK_DRIVER_PYTHON=/opt/conda/miniconda3/bin/python

Copy the jar to the right location

cp mleap-databricks-runtime-fat-assembly-0.22.0.jar $SPARL_HOME/jars echo "export JAVA_HOME=\"/usr/lib/jvm/java-11-openjdk-amd64\"" >> $SPARK_HOME/conf/spark-env.sh `