search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
455 stars 34 forks source link

Java 8 support #183

Closed hawkaa closed 1 year ago

hawkaa commented 1 year ago

Hi @Jolanrensen , and thank you for maintaining this great library!

I'm sorry, but the world that I live in is a world of the past. We, therefore, require Java 8 support in order to use this library.

 What went wrong:
Could not determine the dependencies of task ':jobs:compileJava'.
> Could not resolve all task dependencies for configuration ':jobs:compileClasspath'.
   > Could not resolve org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1.
     Required by:
         project :jobs
      > No matching variant of org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 was found. The consumer was configured to find an API of a library compatible with Java 8, preferably in the form of class files, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' but:
          - Variant 'apiElements' capability org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 declares an API of a library, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm':
              - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8
          - Variant 'runtimeElements' capability org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 declares a runtime of a library, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm':
              - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8

Is there any chance the project can be built with Java 8 support? I'm not super familiar with the JDK/JREs and I'm unsure which consequences it brings.

Håkon

Jolanrensen commented 1 year ago

I think we had some issues with dependencies on Java 8, but I'll investigate again :)

Jolanrensen commented 1 year ago

Hmm, yeah still we get something comparable to this issue https://youtrack.jetbrains.com/issue/KT-16493 when running with java 8. I don't see an obvious solution for this, but feel free to do some trial and error yourself :). The jupyter module needs to have java 11 in any case, but in theory, the rest should be able to run with java 8. That is, when we don't get that error. I was trying it out in this branch: https://github.com/Kotlin/kotlin-spark-api/tree/java8

@asm0dey Do you have any experience with this OutOfMemoryError when running the project with java 8?

hawkaa commented 1 year ago

@Jolanrensen thank you so much for checking this out.

I spent quite a lot of time and I found an upgrade path to Java 11 for us, which seems to work. Sorry for bothering you and thanks again!