search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
459 stars 35 forks source link

Documentation on how to use UDFs #142

Closed vegarsti closed 2 years ago

vegarsti commented 2 years ago

Thanks for maintaining this project!

I would like to use the UDF functionality provided here. I'm, however, struggling with understanding how to use it. I think the UDFWrappers are what I want.

Is there some code somewhere that uses these? I'd be happy to contribute with a PR with for example some tests, when I figure out how to use them.

vegarsti commented 2 years ago

I'm very sorry, there was of course the UDFRegisterTest: https://github.com/JetBrains/kotlin-spark-api/blob/spark-3.2/kotlin-spark-api/3.2/src/test/kotlin/org/jetbrains/kotlinx/spark/api/UDFRegisterTest.kt

Jolanrensen commented 2 years ago

An example inside https://github.com/JetBrains/kotlin-spark-api/tree/spark-3.2/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples would still be valuable I think :) Thanks for letting us know!

Jolanrensen commented 2 years ago

Also look at UDAF https://spark.apache.org/docs/latest/sql-ref-functions-udf-aggregate.html

Jolanrensen commented 2 years ago

Added a complete new notation, support and examples for UDFs in https://github.com/Kotlin/kotlin-spark-api/pull/152. Will be present in the next update :)

https://github.com/Kotlin/kotlin-spark-api/tree/main#user-defined-functions