search

linkedin / transport

A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.
BSD 2-Clause "Simplified" License
297 stars 73 forks source link

UDF Spark registration in Scala is not shown as a method #82

Closed lbrandes3 closed 3 years ago

lbrandes3 commented 3 years ago

When trying to register one of the example UDFs packaged in Transport in a local spark image as described in the Using Transport UDFs doc, after importing the UDF, Scala gives the error that register isn't a member of the object.

When listing all of the methods available to Scala from the class, register isn't listed as one of the methods of the class. It did successfully import though, as the class is available to reference.

I think that this could be related to importing the wrong file, as the classpath is a bit odd -- but it does refer to an actual class that Scala successfully imports. (Classpath: com_linkedin_transport_transportable_udfs_example_udfs_unspecified.com.linkedin.transport.examples.NumericAddIntFunction)

lbrandes3 commented 3 years ago

It turns out that this issue was related to me using the incorrect version of Scala. We were able to solve this by building with Scala 2.11 instead of Scala 2.12.