How to support Spark 2.x

[qiuzhanghua]

when I run Spark 2.2.1 pre-build with Hadoop 2.7 , it failed.

java.lang.IllegalStateException: Failed to execute CommandLineRunner
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:735) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    at org.springframework.cloud.task.app.spark.client.SparkClientTaskApplication.main(SparkClientTaskApplication.java:29) [classes!/:1.3.0.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [spark-client-task-1.3.0.RELEASE.jar:1.3.0.RELEASE]
Caused by: java.lang.RuntimeException: Spark Application failed
    at org.springframework.cloud.task.app.spark.client.SparkClientTaskConfiguration$SparkAppClientRunner.run(SparkClientTaskConfiguration.java:90) ~[spring-cloud-starter-task-spark-client-1.3.0.RELEASE.jar!/:1.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.8.RELEASE.jar!/:1.5.8.RELEASE]
    ... 14 common frames omitted
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
    at org.apache.spark.examples.JavaSparkPi.main(JavaSparkPi.java:36) ~[spark-examples_2.11-2.1.1.jar:2.1.1]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ~[spark-core_2.10-1.6.3.jar!/:1.6.3]
    at org.springframework.cloud.task.app.spark.client.SparkClientTaskConfiguration$SparkAppClientRunner.run(SparkClientTaskConfiguration.java:87) ~[spring-cloud-starter-task-spark-client-1.3.0.RELEASE.jar!/:1.3.0.RELEASE]
    ... 15 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_162]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_162]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_162]
    ... 26 common frames omitted

[qiuzhanghua]

java -jar ~/.m2/repository/org/springframework/cloud/task/app/spark-client-task/1.3.0.RELEASE/spark-client-task-1.3.0.RELEASE.jar --spark.app-class=org.apache.spark.examples.JavaSparkPi --spark.app-jar=examples/jars/spark-examples_2.11-2.1.1.jar --spark.app-args=10

[sabbyanandan]

Hi, @qiuzhanghua. The current release is compatible with spark-core_2.10. If you'd like to upgrade it, you can change the dependency and follow the build procedure to build it locally.

Also, it'd be great if you can contribute it back to the project. Thanks for your interest!

[qiuzhanghua]

Hi, @sabbyanandan .Please see https://github.com/qiuzhanghua/spark-client , I try spark 2.2.1 successfully. Sorry I don't have more time merge it to your project structure.

[sabbyanandan]

Thanks! We will take a look.