Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
456 stars 34 forks source link

Seq creation function #163

Closed Jolanrensen closed 2 years ago

Jolanrensen commented 2 years ago

For creating custom SparkSQL Expressions that use org.apache.spark.sql.catalyst.expressions.LambaFunction and possibly in other places in the API a Scala Seq. Note that the Java API almost never uses Seqs, so this function would only have to be used in edge cases.

Implementation could look like:

fun <T> seqOf(vararg elements: T): scala.collection.Seq<T> = scala.collection.JavaConverters.asScalaBuffer(elements.asList())
tianyu commented 2 years ago

If I may add a few requests:

Jolanrensen commented 2 years ago

Hmm, I'm doubting whether all those extra features would be in the scope of the API... The goal is to make Spark work great from Kotlin, not to improve Scala-Kotlin interoperability per se.

We decided to add all the helper stuff for Scala Tuples because the Java Spark API explicitly requests Tuples and there are speed benefits for using Tuples all the way through instead of converting other classes to them. However, Seqs are not needed for the Java API apart from some very specific edge cases. So while a seqOf() function would be in the scope, all those extras might be better placed in a separate library.

asm0dey commented 2 years ago

@tianyu I'm sorry, but it's definitely not on the plate right now. Thing is our goal is to provide idiomatic Kotlin API, not Scala API. Destructuring this way is alien to Kotlin, for Kotlin destructuring of any list-like structure returns elements.

However everything you want is definitely possible to implement as extension methods, for example

object Nil
fun seqOf() = Nil

Or for destructuring

fun Sequence.component0() = head()
fun Sequence.component1() = tail()

And so on.

Jolanrensen commented 2 years ago

I did btw find a better "direct" translation of Scala's Seq(a, b, c) function which includes Nil in the empty() function:

fun <T> seqOf(vararg elements: T): Seq<T> = with(`Seq$`.`MODULE$`) {
    if (elements.isEmpty()) {
        empty<T>() as Seq<T>
    } else {
        newBuilder<T>().apply {
            for (elem in elements)
                `$plus$eq`(elem)
        }.result()
    }
}
Jolanrensen commented 2 years ago

added for 1.2.0