search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
459 stars 35 forks source link

Update spark to 3.3.2 #198

Closed devongleeson closed 1 year ago

Jolanrensen commented 1 year ago

Awesome! I suppose an update to 3.3.2 from 3.3.1 won't cause many issues. I did change your PR to target main instead of release, as that's common for this project.

Please also update gradles.properties to Spark 3.3.2 as the project always targets the latest possible Spark version by default. Same with the integration examples. I see you found a couple of versions that we forgot to update, thanks :)

I'll try and run the github actions that will perform the tests in a bit :)

Edit: the tests seem to work :)

devongleeson commented 1 year ago

Question: should this be updated? https://github.com/Kotlin/kotlin-spark-api/blob/release/core/build.gradle.kts#L30

Looks like that logic lives in a few places in the project's build.

Jolanrensen commented 1 year ago

Question: should this be updated? https://github.com/Kotlin/kotlin-spark-api/blob/release/core/build.gradle.kts#L30

Looks like that logic lives in a few places in the project's build.

Looks like the tests pass for 3.3.2, so it seems that Spark fixed the issue. I do suspect the issue is still present for 3.3.1, so it can remain like that.

Jolanrensen commented 1 year ago

Thanks for the help! I'll probably publish 1.2.4 soon with spark 3.3.2 support and java 8 support for jupyter notebooks.