We are adding PySpark option to the add-ons flow for framework and it will utilise the spaceflights-pyspark starter as a template. Therefore, we need to add the post_gen_project.py script to the starter to help setup the PySpark template.
Development Notes
Adding the script breaks the CI because develop branch has all the add-ons flow changes not main. Resolved this in https://github.com/kedro-org/kedro-starters/pull/159 where we switched to using kedro develop branch in test_requirements.txt.
The post_gen_project.py here is slightly different to the core template. Here we need to check if PySpark, option 6, is selected before we run the post_gen_project script. This ensures that this script is only run using the add-ons flow and not with kedro new --starters.
Checklist
[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Motivation and Context
Related to: https://github.com/kedro-org/kedro/pull/3169
We are adding PySpark option to the add-ons flow for framework and it will utilise the
spaceflights-pysparkstarter as a template. Therefore, we need to add thepost_gen_project.pyscript to the starter to help setup the PySpark template.Development Notes
Adding the script breaks the CI because develop branch has all the add-ons flow changes not main. Resolved this in https://github.com/kedro-org/kedro-starters/pull/159 where we switched to using kedro
developbranch intest_requirements.txt.The
post_gen_project.pyhere is slightly different to the core template. Here we need to check if PySpark, option 6, is selected before we run thepost_gen_projectscript. This ensures that this script is only run using the add-ons flow and not withkedro new --starters.Checklist