Missing org/apache/spark/internal/Logging$class

[asafcombo]

environment

I'm using emr 6.0.0 with spark 2.4.4 installed.

error

1. spark-shell --jars "/home/hadoop/kinesis-sql/target/spark-sql-kinesis_2.11-1.1.5_spark-2.4.jar"

2. val kinesis = spark 
   .readStream 
   .format("kinesis") 
   .option("streamName", "Test") 
   .option("endpointUrl", "https://kinesis.us-east-1.amazonaws.com")
   .option("region", "us-east-1") 
   .option("startingposition", "TRIM_HORIZON")
   .load()

returns

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.kinesis.KinesisSourceProvider could not be instantiated
  at java.util.ServiceLoader.fail(ServiceLoader.java:232)
  at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
  at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
  at scala.collection.Iterator.foreach(Iterator.scala:941)
  at scala.collection.Iterator.foreach$(Iterator.scala:941)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
  at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
  at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
  at scala.collection.TraversableLike.filter(TraversableLike.scala:347)
  at scala.collection.TraversableLike.filter$(TraversableLike.scala:347)
  at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
  at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161)
  ... 49 elided
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
  at org.apache.spark.sql.kinesis.KinesisSourceProvider.<init>(KinesisSourceProvider.scala:38)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at java.lang.Class.newInstance(Class.java:442)
  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
  ... 66 more

things I already tried

1. download /home/hadoop/spark-core_2.11-1.5.2.logging.jar"
2. spark-shell --jars "/home/hadoop/kinesis-sql/target/spark-sql-kinesis_2.11-1.1.5_spark-2.4.jar,/home/hadoop/spark-core_2.11-1.5.2.logging.jar"

Error persists.

[itsvikramagr]

@asafcombo - can you check the scala version in EMR? kinesis-sql is using scala 2.11? Can you reach out with the EMR support team for resolving your issues?

[asafcombo]

for EMR 6.0.0 and spark 2.4.4 the scala version is 2.12.10

For scala 2.11 the latest spark version that can be used is 2.4.0 which correlates to EMR 5.23.0

[itsvikramagr]

@asafcombo - this library is supported only for scala 2.11. Feel free to start a PR for adding the support of scala 2.12.