Make A Kedro Project Day

[ankatiyar]

Introduction

Tentative date - 9th March, 2023

The purpose of this day is for the team to get together and create our own Kedro projects to get familiar with the users experiences and pain points.

The plan

The idea is for us as a team to try out all the different components of Kedro right from the instantiation of a project to deployment and make note of any bugs or pain points we face along the way and also document the manual testing steps.

I and @AhdraMeraliQB have made a list of some components to test out with our projects. The list is not exhaustive, please feel free to add more to it -

Instantiating a project

• Using kedro new
• Using kedro new with a starter
• Using kedro new with config
• From a packaged project

Project development with the interactive workflow

• Connecting the standalone-catalog to a jupyter notebook
• Using the %load_ext command in a notebook (not using the Kedro commands)
• Using Kedro through the api (as a library)

Scaling project development

• Using micropackages
• Using modular pipelines

Running

• Using different runners
• Using a custom runner
• kedro run from config
• Running sliced pipelines (--from-nodes, --to-nodes, --from-inputs, --to-outputs, --tags, --nodes)

Setting up DevOps

• Linting
• Testing
• Documentation with Sphinx

Deployment

• Packaging a project
• Deploy to Airflow using kedro-airflow
• Any option outlined in the deployment docs

Extras for Advanced Users

• Accessing datasets from external locations (e.g Azure Cloud)
• Modifying configuration with OmegaConf
• Modifying logging
• Using different config environments
• Using hooks
• Using kedro-docker to run Kedro projects in a container
• Using versioned datasets and kedro-viz
• Providing credentials to a project (env vars, credentails.yml, hooks)
• Using a custom dataset
• Adding or modifying CLI commands

Resources

Check out Kaggle Competitions for project ideas. Here's some examples -

• [BINARY CLASSIFICATION] Spaceship Titanic
• [BINARY CLASSIFICATION] Titanic Machine Learning from Disaster
• [MULTI-CLASS CLASSIFICATION] Digit Recognizer
• [REGRESSION] House Prices

Some places to search for datasets for your projects -

• Datasets on Kaggle
• UCI Machine Learning Repository
• Awesome Public Datasets Repo on Github
• Data.gov for data from US government agencies or Data.europe.eu for data from the EU
• The r/datasets subreddit

Bonus : Interesting datasets I found -

• New York Squirrel Census
• UFO Sightings
• World Happiness Index
• Harry Potter Dataset
• New York Times Caption Contest dataset
• Google's Quick! Draw dataset
• Netflix Movie and TV shows

The goal for each project would be to touch on at least one item from each area. At the end of the day there will be time for participants to talk about their projects and their experience creating them.

[datajoely]

❤️ this

[stichbury]

I love this, thank you for the initiative. I know @astrojuanlu and I will be all over this from a devrel perspective. It would be great, for example, to collect all the projects together and I can write some narrative about the day for a blog post and point to them all. Then later we can pull out different aspects into tutorials and posts. Nice one!

Would the kedro-community repo be a good place for us to work on the day, so we have a set of the projects kept together as we build them?

[stichbury]

Would databricks be something to include too?

[merelcht]

This looks great! I'm very excited about this day already 😄 Do you have any thoughts on the projects we could do? E.g. any kaggle competitions or specific datasets that would be good to work with? Having a pre-selected list might help getting started quickly on the day.

[stichbury]

I was wondering if we could use a databricks tutorial as a starting point, and basically rework it to use Kedro on top of Databricks? I found this tutorial example with the million song database in their docs and perhaps we could adapt it, but I've no idea if I'm on the right track here. Maybe something @jmholzer has already seeen/tried?

[astrojuanlu]

Other datasets and interesting sites:

• https://zindi.africa/
• https://www.drivendata.org/
• https://openrepair.org/news/open-repair-alliance-dataset-increases-to-81000-records-of-community-repair/

[stichbury]

An easy one to update (which isn't really creating a new project, more bringing an old one up to date) would be Lais' project on penguins:

https://github.com/laisbsc/kedro_penguins