Add `spaceflights-pyspark` starter - Githubissues

kedro-org / kedro-starters

Templates for your Kedro projects.

Apache License 2.0

63 stars 57 forks source link

Add `spaceflights-pyspark` starter #147

Closed merelcht closed 9 months ago

merelcht commented 10 months ago

Motivation and Context

https://github.com/kedro-org/kedro/issues/2984 subtask of https://github.com/kedro-org/kedro/issues/2838 which in turn is part of the new project creation flow.

How has this been tested?

Created a project locally using: kedro new --starter=/Users/merel_theisen/Projects/kedro-starters/spaceflights-pyspark/ and did kedro run

Things to note:

⚠️ I had to drop columns from all input datasets to make the starter run for me locally. I added a new node to preprocess the reviews input data.
I moved logging.yml outside of the base folder to the top-level conf/ folder
Update the README.md to remove mentions of deprecated commands.
Added TestDataScienceNodes to demonstrate how you can write a unit test for a node.

Questions:

@amandakys / @yetudada Do we intend to add these starters to our official starters so that users could also do kedro new --starter=spaceflights-pyspark?

Checklist

[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Assigned myself to the PR
[ ] Added tests to cover my changes

noklam commented 9 months ago

Will this be introduced before 0.19?

noklam commented 9 months ago

I took a quick look and it's fine. I don't have time to test it out, it's best to have another engineer to approve this maybe @AhdraMeraliQB ?