search

MrPowers / bebe

Filling in the Spark function gaps across APIs
50 stars 5 forks source link

Possibly make a Spark 2 release & talk about maintenance #9

Open MrPowers opened 3 years ago

MrPowers commented 3 years ago

I think we should bump to Spark 2.4.5 & see what can features can get added to Spark 2. This way they'll be some JAR files for the Spark 2 users.

Then think we should bump to Spark 3.0.1 and see what additional features can be added. This is the version for the latest Databricks runtime and would serve current users.

Think the version bumps should roughly keep pace with the Databricks version bumps.

Don't think we should cross compile. We'll be relying on new features that are added every release and don't want to make this a complicated maintenance thing. We can just make it clear what versions are supported for each release so users can easily pick the JAR file that'll work for them.

Thoughts?

alfonsorr commented 3 years ago

Nice to have, but it needs a rework in the sbt part, some of the first SQL functions added here are spark 3.1.0 valid.

It would be great to organize the code in a way that would be easy to maintain in which versions of scala /spark we can have. Also, it will be a good opportunity to start with the CI (github actions I will propose)