search

sourav-mazumder / Data-Science-Extensions

71 stars 63 forks source link

java.lang.ClassNotFoundException: scala.Product$class #11

Open barel166 opened 3 years ago

barel166 commented 3 years ago

I'm running spark locally. Spark v3.0.1 with Scala 2.12.10. I've also made sure that my local version of scala is 2.12. Does the RestDataSource package need to be recompiled for Scala 2.12?

I ran: spark-shell --jars spark-datasource-rest_2.11-2.1.0-SNAPSHOT.jar --packages org.scalaj:scalaj-http_2.12:2.3.0

This is line that yields error:

scala> val weatherDF = spark.read.format("org.apache.dsext.spark.datasource.rest.RestDataSource").options(parmg).load(); java.lang.NoClassDefFoundError: scala/Product$class at org.apache.dsext.spark.datasource.rest.RESTRelation.(RestRelation.scala:41) at org.apache.dsext.spark.datasource.rest.RestDataSource.createRelation(RestDataSource.scala:42) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:344) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:297) at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:286) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:286) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:221) ... 47 elided Caused by: java.lang.ClassNotFoundException: scala.Product$class at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 55 more

sourav-mazumder commented 3 years ago

@barel166 Yes. This code base is not updated for latest spark and scala version for a while.

noodlemind commented 3 years ago

@sourav-mazumder What is the Max Version of Spark / Scala that the code base is most compatible with ?

guruRockz commented 3 years ago

With the following updates to pom.xml(in Data-Science-Extensions folder) and a rebuild, the plugin is working perfectly fine with Spark 3.0.1 (Scala 2.12.10), verified with the example provided by Sourav for spark-shell(scala):

  1. org.scalatest scalatest_${scala.binary.version} **3.0.1** test
  2. org.scalacheck scalacheck_${scala.binary.version} **1.15.2** test
  3. 1.7 2.12.10 2.12 3.3.9 1.7.16 1.2.17 3.0.1
  4. scala-2.12 !scala-2.10 2.12.10 2.12

Attached is the POM(in Data-Science-Extensions folder) with full changes below pom.xml.txt

jonahkaplan1 commented 2 years ago

Hmm, the pom.xml file doesnt seem to be working anymore