Separate python deps

[ghost]

Description

The aim of this PR is to separate the Python dependencies for each pipeline component from each other component and from the ML pipelines themselves. There are a lot of files changed (!) but this mostly consists of files moving and some import statements being updated. The key changes are:

• Pipeline components have been moved out of the pipelines/kfp_components directory and into pipeline_components
• Each group of pipeline components (e.g. aiplatform, bigquery, helpers etc) has its own Python virtual environment managed with pipenv and its own tests in its directory
• The existing pipelines and tests directories are moved to a new top-level directory called pipelines, which also now contains the Pipfile and Pipfile.lock for the ML pipelines virtual env (previously in the root of the repo)
• Instead of importing the @component functions directly for use in ML pipelines, the pipeline components are first compiled to YAML (in their respective virtual environment), before being imported from YAML for use in the ML pipelines
• Updates to python imports, Makefile, CI/CD pipelines and docs related to the above changes

How has this been tested?

• Passing automated tests set up on my fork of the repo - see equivalent PR here
  • The xgboost E2E test is still failing due to #26 - this PR will enable us to fix this more easily by modifying the dependencies for (only) the XGBoost components
• Make commands tested locally

Checklist

• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have successfully run the E2E tests - [ ] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] I have updated any relevant documentation to reflect my changes
• [x] I have assigned a reviewer and messaged them

[simonbutt-datatonic]

Add:

pipeline_components/evaluation/example_count.py 
pipeline_components/evaluation/plots_num_feat.html
pipeline_components/evaluation/plots_overall.html

to .gitignore or clean during specific test. These files are created when running make test-all-components

[simonbutt-datatonic]

pipeline_components/_tfdv/_tfdv/conftest.py if empty, is it needed?

[ghost]

Add:

pipeline_components/evaluation/example_count.py 
pipeline_components/evaluation/plots_num_feat.html
pipeline_components/evaluation/plots_overall.html

to .gitignore or clean during specific test. These files are created when running make test-all-components

Added to .gitignore for this PR. Created #29 to track it and will address properly in a future PR

[ghost]

pipeline_components/_tfdv/_tfdv/conftest.py if empty, is it needed?

Yes this is required for pytest to pick up the directory so that the contents can be imported during tests

[simonbutt-datatonic]

In that case potentially add a comment stating the purpose in the file. Otherwise can image people developing on the template could remove without realising the failing test consequence

[ghost]

In that case potentially add a comment stating the purpose in the file. Otherwise can image people developing on the template could remove without realising the failing test consequence

Added

[ghost]

@Linchin ready for your review :)