search

jpmml / jpmml-evaluator-spark

PMML evaluator library for the Apache Spark cluster computing system (http://spark.apache.org/)
GNU Affero General Public License v3.0
94 stars 43 forks source link

java.lang.NoClassDefFoundError: org.jpmml.model.ImportFilter #20

Closed AbdealiLoKo closed 6 years ago

AbdealiLoKo commented 6 years ago

When trying to use org.jpmml.evaluator.spark.EvaluatorUtil.createEvaluator() I get:

java.lang.NoClassDefFoundError: org/jpmml/model/ImportFilter
    at org.jpmml.evaluator.spark.EvaluatorUtil.createEvaluator(EvaluatorUtil.java:53)
    at org.jpmml.evaluator.spark.EvaluatorUtil.createEvaluator(EvaluatorUtil.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

Seems like EvaluatorUtil.java#L32 is importing org.jpmml.model.ImportFilter but it should be importing org.jpmml.model.filters.ImportFilter

The change occured in https://github.com/jpmml/jpmml-model/commit/d4b7688b14542ccd1a1731e9ac219fbe1f73d6c8 on Feb 10.

vruusmann commented 6 years ago

The JPMML-Evaluator-Spark library inherits from the JPMML-Evaluator library, and the current JPMML-Evaluator-Spark release version 1.1.0 is targeting JPMML-Evaluator version 1.3.11: https://github.com/jpmml/jpmml-evaluator-spark/blob/master/pom.xml#L8

In the 1.3.X development branch, this class is residing in the org.jpmml.model package (you're referring to a commit that happened in the 1.4.X development branch).

In short, it is your Spark application classpath that is broken. Please remove the JPMML-Model version 1.2.15 library from your Apache Spark installation, or apply shading - it's all detailed in the README file.

AbdealiLoKo commented 6 years ago

@vruusmann My understanding of shading was that it is needed only if I built my own jar.

I am just using --packages org.jpmml:jpmml-evaluator-spark:1.1.0,org.jpmml:jpmml-sparkml:1.4.1 which I thought should handle this by itself?

vruusmann commented 6 years ago

@AbdealiJK The --packages org.jpmml:jpmml-evaluator-spark:1.1.0 should work with Apache Spark 2.3.0, which has this JPMML-Model library shading issue resolved (see https://issues.apache.org/jira/browse/SPARK-15526). However, with all earlier Apache Spark 2.X versions (2.0 through 2.2), you need to shade your Spark application as a whole. Sorry, no shortcuts here - and I can't do anything at the JPMML project level to make it work.

Also, I wouldn't advise "activating" JPMML-Evaluator-Spark and JPMML-SparkML together. They depend on different JPMML-Model library versions (the former depends on some 1.3 version, and the latter on some 1.4 version, probably 1.4.1), which can lead to horrible classpath mess.

AbdealiLoKo commented 6 years ago

I am using Spark 2.3.0 - so that is not an issue

I have switched to --packages org.jpmml:jpmml-evaluator-spark:1.1.0,org.jpmml:jpmml-sparkml:1.3.3 to avoid the version mismatch. But I do need to create a PMML and read back that PMML for evaluation - so need both. I'm sure a lot of other folks would need both too...

vruusmann commented 6 years ago

I have switched to --packages org.jpmml:jpmml-sparkml:1.3.3 to avoid the version mismatch.

@AbdealiJK According to JPMML-SparkML version compatibility matrix (https://github.com/jpmml/jpmml-sparkml#library), you would need to use some 1.4.X version with Apache Spark 2.3.0. Earlier versions will throw an IllegalArgumentException stating that there's a platform version mismatch.

AbdealiLoKo commented 6 years ago

You're right. It doesn't work together =(