Implement Spark library

Acxiom / metalus

This project aims to make writing Spark applications easier by abstracting the effort to assemble the driver into reusable steps and pipelines.

Apache License 2.0

15 stars 11 forks source link

Open dafreels opened 1 year ago

dafreels commented 1 year ago

Get project running stand-alone
Provide a step that allows starting a job from a non-spark pipeline
Enable context sharing between calling pipeline and spark pipeline
- SessionContext can be used to restore state one the Spark job starts.
- SessionContext can be used to restore the Spark state once complete for use by the rest of the Pipeline