AlexIoannides / pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.
1.65k stars 701 forks source link

Enhancement/pipenv spark setup #6

Closed AlexIoannides closed 6 years ago

AlexIoannides commented 6 years ago

Switched to using Pipenv for managing dependencies and as part of the dependencies build process, and incorporated contextual awareness in the start_spark() function.