search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
456 stars 35 forks source link

Enhancement: Improved broadcasting #82

Closed Jolanrensen closed 3 years ago

Jolanrensen commented 3 years ago

Instead of calling 

spark.sparkContext.broadcast(

you can now simply call

spark.broadcast(

which matches some of the other helper functions in the API.

Aside from that there are a lot of cases where the encoder<>() function cannot find the encoder for a certain type like Array<DoubleArray> for instance. For these cases I added a fallback on the JavaSparkContext.broadcast function already present in Spark. This makes sure that at the encoder<>() method is attempted (so broadcasting Data classes still works fine), but if that fails it still has a backup solution.

Tests are included of course.

asm0dey commented 3 years ago

Please, let's preserve old API too just not to break compatibility, but I like new API more!