General refactoring and cleanup of the repo

CoVital-Project / Spo2_evaluation

Python script to evaluation the correctness of SpO2 estimation algorithms

18 stars 6 forks source link

General refactoring and cleanup of the repo #20

Open gianlucatruda opened 4 years ago

gianlucatruda commented 4 years ago

TODO:

[x] Update requirements.txt (see PR #22)
[ ] Improved documentation in data_loader and preprocessing
[x] Add a proper .gitignore file (see PR #22)
[x] Structure the first level of project directories (see PR #22)
[x] Remove video_data files from LFS and store as a .zip on Dropbox (see PR #22)

gianlucatruda commented 4 years ago

Proposed project structure to make the repo more modular and accessible:

Spo2_evaluation/
├── README.md
├── evaluation
│   ├── README.md
│   ├── Spo2_evaluation.py
│   ├── blant_altman.py
│   ├── test.py
│   └── test_normal.py
├── misc
│   ├── VideoDataAcquisition.java
│   └── fetch_data.py
├── modelling
│   ├── NN_models
│   ├── healthwatcher
│   ├── lamonaca_and_nemcova
│   └── wang_2017
├── preprocessed_data
│   ├── README.md
│   ├── nemcova_data
│   └── sample_data
├── preprocessing
│   ├── README.md
│   ├── data_loader.py
│   ├── structure.py
│   └── utils.py
└── requirements.txt

MalcolmMielle commented 4 years ago

I think we should remove the videos inside the video_data folder from the repo entirely and instead add a bunch of loading scripts in Python so we don't increase the size of the repo too much with videos.

I see multiple solution scripts we could make available:

Local load of the dataset from Dropbox or other provider
Batch load of a subset of a dataset from an online source so that the user doesn't have to download all the data in one go.

gianlucatruda commented 4 years ago

I think we should remove the videos inside the video_data folder from the repo entirely and instead add a bunch of loading scripts in Python so we don't increase the size of the repo too much with videos.

I agree! I'll have a look at ways we could request, unzip, and arrange the data via a simple script.

gianlucatruda commented 4 years ago

I've started reorganising the project in the refactoring branch. PR #22

LittlePea13 commented 4 years ago

Maybe add the NN models in the modelling folder?

gianlucatruda commented 4 years ago

Maybe add the NN models in the modelling folder?

@LittlePea13 Yes! Thanks! They weren't yet merged when I was doing this initially, but I've added them now.

gianlucatruda commented 4 years ago

@MalcolmMielle I've added a script for fetching and unzipping the data files

https://github.com/CoVital-Project/Spo2_evaluation/blob/refactoring/misc/fetch_data.py