Closed Jolanrensen closed 2 years ago
@Jolanrensen thank you! Does this version work with older Spark?
@asm0dey I'm not sure! I haven't tested that yet. I simply updated the versions and fixed like one override
. We don't have any tests that try this, maybe you could look into that?
Although, not much has changed regarding the API, especially since all the tests we do have passed immediately, so I think it might be fine.
I thought about such tests, but it's something non-trivial to implement because it will force us to have somehow tested with different Spark versions.
Yeah exactly. But how would you want to test it then? Use the API with spark 3.2.0 as library in a project that used 3.0.0 before?
Set up a new project built with kotlin-spark built against spark 3.2 and add an explicit dependency on Spark 3.0 to it, I think.
Ah, that seems to give errors. Like Exception in thread "main" java.lang.NoSuchMethodError: 'scala.collection.Seq org.apache.spark.sql.catalyst.expressions.objects.Invoke$.apply$default$5()'
.
This means that the API needs to have the same version of Spark as the end-user will use.
Edit: actually, let me check if the scala version is the same (2.12.15, I was using 14 still) Edit2: No, it's definitely the Spark version. If I make my project target 3.2.0 it works again.
And this automatically means we need a separate spin-off for 3.2 :(
If you still want to support 3.0.0, then yes
What errors does it give when you use current version with 3.2?
I get the same error as when mismatching the versions the other way around. I also read complaints on Slack that it was broken. I'm pretty sure that a cross-version solution is not the best way to go, though. So I'd say, drop 3.0 support for the next version, since 3.2.0 is the last 3.x Spark version anyways.
What I would do is always target the latest version, which now is 3.2, and then one LTS version which currently is 2.4 (according to Wikipedia). That way, the users get pushed to update their stuff, haha.
Sounds reasonable, but it requires some further thinking about the process. Should we bump the major version because it's backward incompatible? And also… what if we find a bug affecting both 3.0 and 3.2 versions? It sounds like we should somehow release the support branch.
Hmm maybe we can make it 1.1.0? As Spark also only changed the second number.
Hmm, yeah if you want to have a support branch for 3.0 still, you can create that, but then it also needs to be maintained when we find a bug. The project, however, isn't very active all the time, so I think only supporting the latest version would be enough. That is, unless there's still a lot of requests for a 3.0 version.
It's up to you, it's your project after all :).
Maybe you're right. It's interesting that folks from Spark broke backward compatibility in a minor release. Maybe we should really bump the version to 1.1.0 and support the old version in the 1.0 branch.
Possibly we should create branch like spark-3.2 here, bump there version and add compatibility with spark 3.2 in this branch. And do releases from there too.
I would choose the option that is the least work for the future so that we can make updates faster when a new scala- or spark update comes around :)
The next Scala is 3 and I think that it'll be a huge piece of work to support it…
Not sure how quickly spark is updated to scala 3 however. I was more aiming towards another scala 2.x.x version.
But I'd say spark 3.2 support should be released as soon as possible anyways as there are people now dropping Kotlin because they cannot use the API on 3.2 yet :/ So we should either drop support for 3.0 or keep it on a separate branch.
@Jolanrensen may I ask you please to do the following: change your fork to work with the spark-3.2 branch, remove support for spark 2 there and change the code to work with 3.2
Sure, this will also remove scala 2.11 then right? Since that only works with Spark 2
Do you also want to bump to kotlin 1.6.10? or should we do that next
https://github.com/Jolanrensen/kotlin-spark-api/tree/spark-3.2 Alright, here is my branch based on your spark-3.2 brach. Old Scala and Spark 2 is removed and everything is 3.2 now.
I think you can just create aPR against my branch!
Alright, moving to https://github.com/JetBrains/kotlin-spark-api/pull/118
Updated to Spark 3.2.0 and Scala 2.12.15. Tests seem to work