Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
456 stars 34 forks source link

SparkSession ClassNotFoundException gradle kts #145

Closed Icyrockton closed 2 years ago

Icyrockton commented 2 years ago

Hi! I am new to kotlin spark , also new to spark , playing around with sample project this I get error

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
    at SimpleApp.main(SimpleApp.kt:15)
    at SimpleApp.main(SimpleApp.kt)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:636)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:182)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:519)
    ... 2 more

also I also tried in my own project using the latest version and kotlin 1.6.2 , which get the same error , my gradle kts file

import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar

plugins {
    kotlin("jvm") version "1.6.20"
    id("com.github.johnrengelman.shadow") version "5.2.0"
    scala
}

group = "org.example"
version = "1.0-SNAPSHOT"

repositories {
    mavenCentral()
}

java {
    sourceCompatibility = JavaVersion.VERSION_11
    targetCompatibility = JavaVersion.VERSION_11
}

dependencies {
    implementation(kotlin("stdlib"))
    implementation("org.jetbrains.kotlinx.spark:kotlin-spark-api-3.2:1.0.3")
    implementation("org.scala-lang:scala-library:2.12.15")
    compileOnly("org.apache.spark:spark-sql_2.12:3.2.1")
}

tasks {
    build {
        dependsOn(shadowJar)
    }
}

tasks{
    named<ShadowJar>("shadowJar"){
        dependencies {
            exclude {
                it.moduleGroup == "org.apache.spark" || it.moduleGroup == "org.scala-lang"
            }
        }
    }
}
Icyrockton commented 2 years ago

fixed by change compileOnly to implementation the documentation should update 😂

dependencies {
    implementation(kotlin("stdlib"))
    implementation("org.jetbrains.kotlinx.spark:kotlin-spark-api-3.2:1.0.3")
    implementation("org.apache.spark:spark-sql_2.12:3.2.0")
}