cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
158 stars 89 forks source link

Added Service for Storing/Retrieving Broadcasts #18

Closed rickysaltzer closed 7 years ago

rickysaltzer commented 7 years ago

Introduced BroadcastManager class to allow the user to store broadcasts made using Spark for later use.

BroadcastManager<ComplexRecord> recordManager = new BroadcastManager<>();
Broadcast<ComplexRecord> complexRecordBroadcast = sparkContext.broadcast(complexRecord);
recordManager.put("abc", complexRecordBroadcast);