EDA + Data Analysis Issue (fix feedback issues form Milestone 1)

[sanash43]

• [x] * read data directly from source instead of github repository
• [x] * create train-test split before eda and data analysis
• [x] * obtain feature importances
• [x] * Revise Abstract, Discussion/Conclusion
• [x] * Fix README.md for container instructions

[sanash43]

All these changes are in the Milestone1.ipynb file in the project folder, have implemented changes to the following scripts:

• read_data.py
• data_split.py
• cleaning.py
• eda_figures.py
• preprocessing.py
• STILL NEEDS CHANGING: analysis.py

[sanash43]

To test the scripts that have already been fixed:

make sure you're running the container: docker-compose pull docker-compose run --rm project-image bash

Your command line should look like: jovyan@5c76e9165ffb:

Navigate to the src folder: `cd src

To test the scripts, enter the following into the command line in order:

• read_data.py: python read_data.py 468 ../data/raw_features.csv ../data/raw_targets.csv
• data_split.py: python data_split.py ../data/raw_features.csv ../data/raw_targets.csv ../data/x_train.csv ../data/x_test.csv ../data/y_train.csv ../data/y_test.csv
• cleaning.py: python cleaning.py ../data/x_train.csv ../data/y_train.csv ../data/cleaned_train_data.csv
• eda_figures.py: ```
• eda_figures.py: python eda_figures.py ../data/cleaned_train_data.csv figure
• preprocessing.py: python preprocessing.py ../data/x_train.csv ../data/x_test.csv ../data/y_train.csv ../data/y_test.csv ../data/preprocessed_train_data.csv ../data/preprocessed_test_data.csv

[nourshawk]

updated commands to test the scripts:

read_data.py: python read_data.py 468 ../data/raw_features.csv ../data/raw_targets.csv cleaning.py: python cleaning.py ../data/x_train.csv ../data/y_train.csv ../data/cleaned_features.csv ../data/cleaned_targets.csv data_split.py: python data_split.py ../data/cleaned_features.csv ../data/cleaned_targets.csv ../data/x_train.csv ../data/x_test.csv ../data/y_train.csv ../data/y_test.csv eda_figures.py: python eda_figures.py ../data/cleaned_features.csv ../data/cleaned_targets.csv ../img/figure preprocessing.py: python preprocessing.py ../data/x_train.csv ../data/x_test.csv ../data/y_train.csv ../data/y_test.csv ../data/preprocessed_train_data.csv ../data/preprocessed_test_data.csv analysis.py: python analysis.py ../data/preprocessed_train_data.csv ../data/preprocessed_test_data.csv ../results

[sanash43]

all scripts and makefile have been fixed, quarto report runs fine now