search

abronte / PysparkProxy

Seamlessly execute pyspark code on remote clusters
Other
4 stars 0 forks source link

Initial UDF support #19

Closed abronte closed 6 years ago

abronte commented 6 years ago

Investigate what limitations there might be with UDFs and see if simple UDFs can be implemented.

def squared(s):
  return s * s
sqlContext.udf.register("squaredWithPython", squared)
abronte commented 6 years ago

Going to ignore the spark.udf.register() use case for now since no SparkSession class is mocked out.