search

jpmml / jpmml-evaluator-spark

PMML evaluator library for the Apache Spark cluster computing system (http://spark.apache.org/)
GNU Affero General Public License v3.0
94 stars 43 forks source link

local class incompatible #34

Closed zhuzhujulie closed 4 years ago

zhuzhujulie commented 4 years ago

hi dear author, i use your ilbrary to contrust a predict from hdfs data to hdfs data, but in the final step, i use dataframe.write.format("...csv").path("/../") to save data into hdfs. then i got an error messaeg said "java invalidClassException: org.jpmml.evaluator.ModelEvaluator,local class incompatible, serialVersionUiD=blabla.. , local serialVersionUID=blabla..." i search solutions from internets and take some methods, first i change the pom.xml in order to get the jar version of cdh6.1.1 , because my spark environment is based on cdh6.1.1, then it doesn't work. next, i use the maven-shade-plugin to change the org.jpmml into org.shaded.jpmml, but it still doesn't works. i don't know how to solve this problems, i would appreciate if you can give me some suggestions to help me find the problems... thank you so much~

vruusmann commented 4 years ago

i use the maven-shade-plugin to change the org.jpmml into org.shaded.jpmml, but it still doesn't works.

Use the default JPMML-Evaluator-Spark distributable, as published in the Maven Central repository. Don't change anything about it locally, especially don't change its JPMML-Model transitive dependency.

Right now, you're sending an org.jpmml.evaluator.ModelEvaluator object instance from one JVM to another JVM using the Java serialization mechanism, and it cannot be done, because those two JVMs use different JPMML-Evaluator library versions.

Fix your classpath. Start with actually reading what's written in the README file.

zhuzhujulie commented 4 years ago

hi dear author, whar do you mean of "fix my classpath", my local jdk is verson8 and in spark the jdk is also version8, is there any other things i need to fix? i still can't fix this problem , another question is , i read your read.md , i think maybe i can use your maven repository 1.2.2 jar release to construct my project ranther than using your jar code directly, but i dont't understand your code. for example, what does the class "LoadingModelEvaluatorBuilder" come from? in fact i don't know how to keep the jar version consistently of my local environment and spark environment, can you tell my some metods?