tests
directory Data wraggling/inital exploration notebooks
goes under experiments
directoryanalysis_framework
, create subdirectories as required utils
, data
and pipelines
utils
reusable code componentsdata
raw dataset are stored pipeline
dagster pipeline and related computation output
# for the first time
git clone <repo>
cd
### Installing env setup using miniconda 💁🏽♀️
- [Instructions](https://docs.conda.io/en/latest/miniconda.html)
### Commands to install the packages via conda/ env activation ✍️
```shell
conda env create --file environment.yml
conda activate capstone
conda list
conda info
pip install -e .
capstone
environment (post making changes to environment file)conda env update --file environment.yml --prune
conda deactivate
Just like the name suggests, precommit-hooks are designed to format the code based on PEP standards before committing. More details 🗒
pip install pre-commit
pre-commit install
"""---commit your changes---
- check for errors, hooks format the code by deafult
- add the files
- commit it again
- push the changes
"""
Dagster
, an open source orchestration tool Commands to trigger the pipeline
capstone_isb
python analysis_framework/pipeline/process.py
dagit -f analysis_framework/pipeline/process.py
app.py
(panel serve requirements and layout components) and base.py
(contains all the logic)python analysis_framework/dashboard/app.py