Vectorized UDF: Scalable Analysis with Python and PySpark - Githubissues

manuzhang / read-it-now

Don't read it later; read it now

4 stars 0 forks source link

Vectorized UDF: Scalable Analysis with Python and PySpark #26

Open manuzhang opened 6 years ago

manuzhang commented 6 years ago

Slides

https://www.slideshare.net/databricks/vectorized-udf-scalable-analysis-with-python-and-pyspark-with-li-jin

Notes

PySpark allows for user defined Python functions with the cost of serialization between Java and Python processes

Pandas UDF introduced in Spark 2.3 tries to eliminate the overhead with Apache Arrow and vectorization
Summary of Pandas UDF compared with existing UDF