Large Scale Feature Aggregation Using Apache Spark

[manuzhang]

Slides

• https://www.slideshare.net/databricks/large-scale-feature-aggregation-using-apache-spark-with-pulkit-bhanot-and-amit-nene

Notes

• Feature aggregation is actually time series aggregation on various sizes of windows which can not be solved by batch or streaming systems alone.

• The solution is a lambda architecture, using Spark for long-window aggregation and Athena (Streaming) for short-window, and merging them for real-time access.