java.lang.NoSuchMethodError: org.jpmml.sparkml.SparkMLEncoder.getDataField

[prz999]

Hello! I'm currently trying to generate a PMML file by following the example application you provided. However, I encountered an error message below when executing a similar command on my Spark cluster. I was wondering if you could offer any helpful guidance or resources that could assist me in resolving this issue. Thank you in advance for any assistance you can provide.

Exception in thread "main" java.lang.NoSuchMethodError: org.jpmml.sparkml.SparkMLEncoder.getDataField(Ljava/lang/String;)Lorg/dmg/pmml/DataField;
        at org.jpmml.sparkml.SparkMLEncoder.getFeatures(SparkMLEncoder.java:99)
        at org.jpmml.sparkml.feature.VectorAssemblerConverter.encodeFeatures(VectorAssemblerConverter.java:43)
        at org.jpmml.sparkml.FeatureConverter.registerFeatures(FeatureConverter.java:52)
        at org.jpmml.sparkml.PMMLBuilder.build(PMMLBuilder.java:117)
        at org.jpmml.sparkml.example.Main.run(Main.java:219)
        at org.jpmml.sparkml.example.Main.main(Main.java:159)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[vruusmann]

I'm currently trying to generate a PMML file by following the example application you provided.

Did my example application work?

However, I encountered an error message below when executing a similar command on my Spark cluster.

There is something fundamentally wrong with your Spark cluster classpath. Most likely, there are some base-level JPMML conversion libraries missing.

Please note that JVM is attempting to resolve a SparkMLEncoder#getDataField(String) method.

There is no such method at org.jpmml.sparkml.SparkMLEncoder class level. This method is declared in its superclass, at the org.jpmml.converter.PMMLEncoder class level: https://github.com/jpmml/jpmml-converter/blob/1.5.4/pmml-converter/src/main/java/org/jpmml/converter/PMMLEncoder.java#L96-L98

The JVM method resolution chain fails because you don't have the JPMML-Converter library on your Spark cluster classpath.

I was wondering if you could offer any helpful guidance or resources that could assist me in resolving this issue

Your application classpath contains the JPMML-SparkML library, but it does not contain its essential transitive dependencies (such as JPMML-Converter, JPMML-LightGBM, JPMML-XGBoost etc).

Please revisit the "Installation" section in README.

The easiest thing to do is to use one of the pre-built JPMML-SparkML-Example uber-JAR files, as available under the "Releases" page: https://github.com/jpmml/jpmml-sparkml/releases