Model Evaluation - Githubissues

miriamkw commented 6 months ago

GluPredKit refactor for enhanced and strict model evaluation:

Refactor:

[x] All models should be multiple output
- [x] Create multioutput models
  - [x] Scikit models
  - [x] Neural networks
- [x] Create multioutput preprocessor
- [x] Update all CLI commands to support multiple outputs
- [x] Delete outdated files
[x] What-if events should be default
[x] Fix plug-and-play error: ohio should also be able to be used with normal preprocessor
[x] Update README to the refactoring

Configuration:

[ ] Number of time lagged events should be specified for each feature (list instead of integer)
[x] Configuration has one prediction horizon
[x] Test size should be removed to parsers, CLI command instead. In that way, we can make preprocessors properly plug-and-play
[x] Maybe the type of numerical transformation should be an input to the configuration instead of creating a new preprocessor for every time you want to use something different

Generate pdf reports for model evaluation:

[x] Create CLI command to generate a report
[x] Add a text at top or bottom that refers to GluPredKit ("This report is generated using GluPredKit...")
[x] Add dataset information
[x] Add configuration information
[x] Add traditional evaluation metrics
[x] Add run time evaluation
[x] Insulin and carbohdyrate effect evaluation: Use PCP or the other number (S..). Idea for Partial Dependence Plot: x-axis is prediction horizon, y axis is predicted blood glucose, but then we can have many lines with labels like (UI = 1, UI=2...).
[x] Generate a report that compare models - have in mind that datasets usually will contain many subjects.

Models:

[x] Delete models that don't fit our requirements? Or define included models based on inclusion criterion. Maybe models from the literature that have included their source code for example? Use Doyoungs work as help?
[x] Uva padova model
[x] T1D loop model
[x] Tidepool Loop model

Parsers:

[x] Parsers should process data with the new set of features that we agree upon with Sam
[ ] Add OpenAPS as well
[x] Should assume that all the dataset subjects are in the dataset initially
[x] Parsers should already have separated train and test data to avoid data leakage
- [x] Remove test data fraction from configuration
- [x] Add it to parsing CLI command instead?
[x] Make sure that all parsers use the same unit for basal rates

Metrics:

[x] Add MARD number

Testing:

[x] Make sure that all components have tests and that running the tests is working

General improvements:

[x] Be clearer about the expected input and output formats for each module
[x] Add citation information in README

Checklist before merge

[ ] Clean up the code properly
[ ] Make sure that all the CLI commands are working on both Win and Mac
[x] Run tests
[x] Update README to reflect all of the changes

miriamkw commented 6 months ago

Subtasks adding Loop model:

[x] Use the 1800 rule ++ to fit the therapy settings to individuals
[x] Figure out how to handle smoothly the history lengths when calculating metrics: Use the "process data" method to preprocess the data and remove nan values from the dataset if any
[x] Add future carbs and insulin
[x] Make sure that parsers add basals correctly and loop model uses basals correctly wrt units

miriamkw commented 6 months ago

Subtasks adding whole Ohio dataset evaluation:

[x] Parsing: Storing the dataset in one file, store subject id in a column.
[x] Fix the basal rate in the dataset!
[x] Add finger sticks into dataset!
[x] CLI: Parse the whole dataset, store in one file
[x] README: Update the parser command instructions
[x] Make sure that the sorting of the ids is happening correctly so that the predicted values are correctly aligned with the true values.
[x] Make sure that the loop model input looks correct when it is fed into glupredkit in the second preprocessing

miriamkw commented 6 months ago

Subtask adding the single model evaluation:

[x] Add a test-model function to store predictions and configurations so that you can prototype the generated reports more quickly
[x] store predictions for a ridge and a loop model
[x] train a loop model with optimizing parameters for a loop model to make the predictions and distributions more realistic
[x] Model accuracy:
- [x] Add metrics: MARE, ME/MRE
- [x] Plot parkes error grid (pick max 6 within the prediction horizon)
- [x] Write parkes error grid results in a table
[x] Glycemia Detection:
- [x] Hypo: Plot accuracy, MCC across the prediction horizon
- [x] Hyper: Plot accuracy, MCC across the prediction horizon
- [x] Total: Make a table, calculate overall results
[x] Predicted distribution: Remove the second plot, add indication from the test data, maybe add an interpretation of it in text format? Make it visually clear what is the preferred range.
[x] Physiological Alignment:
- [x] Calculate SHAP numbers. They provide positive or negative impact indication, and magnitude.
- [x] Plot a matrix of SHAP numbers across the prediction horizon and for different time lags
- [x] Add color coding to the matrix so we can clearly see the impact.
- [x] Make the color red if it contradicts the physiological reality
- [x] Make an interpretation of the matrix in writing? Or add the plot for a physiological model.
[x] Finish the first page overview of the model evaluation
[x] Add the linear and exponential error grid metric
[x] Write a document explaining all of the plots and the numbers and the benchmark naive algorithms
[x] Clean up the code so that the cli file is very understandable, and making a helper file that can be more messy

miriamkw commented 6 months ago

Subtask adding the compare models evaluation:

[x] Use the stored results to generate reports
[x] Model accuracy
- [x] Add the naive models, train and test
- [x] Plot and print table RMSE
- [x] Plot and print table ME
- [x] Implement error grid cost functions
- [x] Plot and print table Error grid cost functions
[x] MCC Glycemia Detection
- [x] Add hypo, hyper and total results and raking to table
- [x] Plot hypo
- [x] Plot hyper
[x] Physiological alignment
- [x] Add carbs, insulin and total results and ranking to table
- [x] Plot carbs
- [x] Plot insulin
[x] Predicted distribution
- [x] Plot percentages of the measured distribution for predicted
- [x] Add a table with average results and ranking
[x] Front page: Make an overview over the rankings and try to compute an overall ranking
[x] Fix the physiological alignment plots: we need a way to eliminate the basal rate from the effect
[x] Add a zero-hold algorithm and store predictions
[x] Add a naive linear regressor and store predictions
[x] Add a 3rd or 4th order arx and store predictions
[x] Benchmark together with zero-hold algorithm, linear regressor, and 3rd or 4th order ARX

miriamkw commented 5 months ago

To do experiments:

[ ] Figure out the right configurations for the different models
[ ] Implement in config different preprocessing methods (feature scaling))
[ ] Proper hyperparameter tuning for each model, and document which ones you use

miriamkw commented 5 months ago

JOSS To dos:

[x] Parsers
- [x] Specify test fraction in CLI
- [x] Remove test fraction from configuration
- [x] Separate into test and train data immediately
- [x] Remove it from configs
- [x] Update README
- [x] Make sure that basal is always in the same format
- [x] Document the parser input and output in README
[x] Preprocessors (think simple solutions!)
- [x] Support multiple outputs
- [x] Support train and test splits
- [x] One preprocessor standardscaling, another does nothing
[x] Models
- [x] Make the nns support multiple outputs
- [x] Document the models properly, add citations if relevant
- [x] Make some of the models flexible with inputs for training model (and update CLI)
[x] Plots and metrics: Use the test_model files instead and make sure they are still functioning
- [x] Make sure that the CLI is updated
[x] CLI
- [x] Make sure the CLI is updated to the new refactor
- [x] Update README
[x] README
- [x] Define input and output formats for each module ("Community Contributions")
- [x] Make sure all the commands reflect the refactor
[x] Finish the generated reports
- [x] Add the commands to the README
[x] Dependencies: Make sure they are not too strict, and that all dependencies are there
[x] Code cleanup: Remove all unneccessary.prints and comments in the code.
[ ] Error testing with both windows (idun!) and mac, and both with pypi and the repo directly
[x] Add tests
- [x] All models should have tests like: predict method output format given a certain configuration
- [x] Fit method takes in different inputs
- [x] parsers should also have tests for the given input and output formats (returning dfs with at least a subset of columns)
- [x] metrics expected outputs for some given inputs
[ ] Add CI (Sam)
[x] Add doyoungs implementations!
[x] Add a "predicting mean values" as a benchmark model
[x] Update the figures
[x] Update the video (don't overengineer!)?
[ ] Merge to main
[ ] Push to PyPi
[ ] Utilize code coverage tool (for example codecoverage.py)
[ ] Check the JOSS checklist

miriamkw commented 5 months ago

JOSS specifics, last to dos:

[x] Write good documentation and tests
[ ] Create a branch from the main branch, and add the paper.md with bib there
[x] Send the paper for review from coauthors
[x] Clean up issues and prs in github
[ ] Check that the paper compiles using github action
[x] README should contain:
- [x] A statement of need: which problems the software is designed to solve, who the target audience is, and its relation to other work
- [x] installation instructions (pip install ...)
- [x] examples of how to use the software
- [x] API documentation: all functions are documented including example inputs and outputs
- [x] Community guidelines: clear guidelines for third-parties wishing to contribute, report issues or problems, or seek support
[ ] Tests: an automated test suite hooked up to continuous integration (github actions, circle CI or similar)
[ ] Sample dataset: There should be a sample dataset because reviewers are expected to install the software and verify core functionality

miriamkw commented 4 months ago

Tests:

[x] Read into unit tests vs integration tests
[x] Finish testing the cli commands, make sure that all the models are running and so on
[x] Make unit tests on specific functions
- [x] Determine which functions are called most often by using a profiler
- [x] Create mock data for tested models etc
- [x] Test those functions
[x] Separate unit tests and integration tests into two folders, and update README to instruct those correctly

miriamkw / GluPredKit

Model Evaluation #58