Spark runs on Java 8/11/17, Scala 2.12/2.13, Python 3.7+ and R 3.5+. the library only support Java 11

[etika-garry]

Hi

As the title states apache spark supports various runtimes but most notably the runtime for java is 8/11/17. Unfortunately in many places you are stuck with a specific version of java runtime, in my case 8. This produces obviously the below error:

Execution failed for task ':compileKotlin'.
> Error while evaluating property 'filteredArgumentsMap' of task ':compileKotlin'
   > Could not resolve all files for configuration ':compileClasspath'.
      > Could not resolve org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1.
        Required by:
            project :
         > No matching variant of org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 was found. The consumer was configured to find an API of a library compatible with Java 8, preferably in the form of class files, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm' but:
             - Variant 'apiElements' capability org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 declares an API of a library, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm':
                 - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8
             - Variant 'runtimeElements' capability org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.0_2.12:1.2.1 declares a runtime of a library, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally, as well as attribute 'org.jetbrains.kotlin.platform.type' with value 'jvm':
                 - Incompatible because this component declares a component compatible with Java 11 and the consumer needed a component compatible with Java 8

[Jolanrensen]

I've made attempts to run the Kotlin Spark API with java 8: https://github.com/Kotlin/kotlin-spark-api/issues/183 However, unfortunately, we have too many dependencies that break on java 8. If you or someone can make it work for java8 I'd gladly add it, since indeed, many Spark applications require 8, however no luck thus far (my attempt: https://github.com/Kotlin/kotlin-spark-api/tree/java8).

Java 17 should be possible in the future for sure!

[etika-garry]

Hi Jolan,

Thank you for your attempt. I would be more than happy to introduce J8 compatibility but unfortunately as I just started in a new role where "python" is the main solution I cannot invest the time.

It may be something I look at when I have a little down time.

Even so this should probably be under a feature request as I do not think J8 support will drop at any time. Further, as you have ambitions to have the kotlin-spark-api as core feature in spark it would have to support j8.

Thanks

Garry

[Jolanrensen]

While the tests still don't work, you can try to take this branch https://github.com/Kotlin/kotlin-spark-api/tree/java8 as is, set the scala/spark version you want in gradle.properties and then publishToMavenLocal. In theory, it should work with java 8. Let me know if it does! In the meantime I'll try to fix Kotest...

[Jolanrensen]

I managed to work around the testing issues! See https://github.com/Kotlin/kotlin-spark-api/pull/189. Will soon merge and a dev version will be built to test.

[Jolanrensen]

https://github.com/orgs/Kotlin/packages?tab=packages&q=kotlin-spark-api 1.2.3-SNAPSHOT can be found here with java 8 support :) Feel free to try!

[Jolanrensen]

https://github.com/Kotlin/kotlin-spark-api/releases/tag/1.2.3

[etika-garry]

Hi

This is fantastic news, I've been away since the 10th December so sorry that I was unable to try it.

Great work

On Fri, 30 Dec 2022 at 00:49, Jolan Rensen @.***> wrote:

I managed to work around the testing issues! See #189 https://github.com/Kotlin/kotlin-spark-api/pull/189. Will soon merge and a dev version will be built to test.

— Reply to this email directly, view it on GitHub https://github.com/Kotlin/kotlin-spark-api/issues/186#issuecomment-1367335205, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4MD6TSBMDMO3FWPODILXCLWPWJHPANCNFSM6AAAAAASK64D7A . You are receiving this because you authored the thread.Message ID: @.***>

--

This e-mail, together with any files transmitted with it, are confidential, and are intended solely for the use of the individual or entity to whom they are addressed. Any unauthorised dissemination or copying of this e-mail or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you have received this e-mail in error please notify the sender  and delete it from your system.

[Jolanrensen]

Let me know if it now works for you!