TresAmigosSD / SMV

Spark Modularized View
Apache License 2.0
42 stars 22 forks source link

call sc.addPyFile on all user libs incase they have udfs defined. #1608

Open AliTajeldin opened 5 years ago

AliTajeldin commented 5 years ago

If a udf exists in the a user lib (e.g. 'library/core.py'), the executor will not be able to import the core library when it tries to deserialize the udf.