search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
459 stars 35 forks source link

Questions about project roadmap and adoption #121

Closed devictr closed 2 years ago

devictr commented 2 years ago

Hi folks,

First of all, thank you for all the work you've done to create this Kotlin Spark API. My company is evaluating using kotlin-spark-api for our Spark pipelines. As you know, when evaluating a new library, we need to ask ourselves if it is being actively maintained and assess any risks of becoming too dependent on it. Thus, please allow me to ask the following questions:

  1. What are the future plans for this project?
  2. What is the current status regarding integrating it inside the official Spark API? I see that the SPIP discussions haven't been active since last July
  3. Is the project officially and actively maintained by JetBrains? Is it also being used internally at JetBrains?

I believe that having answers to those questions would be beneficial to my company, other companies or projects that may be wondering the same things when evaluating kotlin-spark-api, and perhaps help reassure people and increase adoption in the short/medium term.

Thanks!

Jolanrensen commented 2 years ago

Hi there!

I recently joined Jetbrains to help with the Spark API specifically. This means it is now definitely actively maintained :). In the short term, we are looking at streamlining the project so that releasing the API for the latest couple of Spark versions, as well as Scala versions, can be done quicker and more easily. It's namely a bit difficult at the moment. In the long term there are also some ideas for features that are not present yet. Feel free to leave an issue if you think something should be added!

As for the official Spark API, we know as much as you do. Apache probably does not want their project to depend on the Kotlin libraries, so official integration by Apache will likely not be a thing. That said, .NET is also not officially supported by Apache, but there's also a Spark API for that, so if we can keep it as "official" as that, that's good, right?

devictr commented 2 years ago

Hi @Jolanrensen

Congrats on the new gig! That is great to hear. Thank you for all those precisions