search

rvandewater / YAIB

🧪Yet Another ICU Benchmark: a holistic framework for the standardization of clinical prediction model experiments. Provide custom datasets, cohorts, prediction tasks, endpoints, preprocessing, and models. Paper: https://arxiv.org/abs/2306.05109
https://github.com/rvandewater/YAIB/wiki
MIT License
50 stars 9 forks source link

Debug flag broken #138

Closed HendrikSchmidt closed 1 year ago

HendrikSchmidt commented 1 year ago 
icu-benchmarks -db \                                                   
    -d demo_data/mortality24/mimic_demo \
    -n mimic_demo \
    -t BinaryClassification \
    -tn Mortality24 \
    -m LSTM \
    --generate_cache \
    --seed 2222 \
    -l ../yaib_logs/  
2023-08-19 19:56:26 - INFO - root : Running experiment mimic_demo.
2023-08-19 19:56:26 - INFO - root : Task mode: Classification.
2023-08-19 19:56:26 - WARNING - root : --------------------No GPUs available: please check your device and Torch,Cuda installation if unintended.--------------------
2023-08-19 19:56:26 - INFO - root : --------------------------------------Logging to ../yaib_logs/mimic_demo/Mortality24/LSTM.---------------------------------------
2023-08-19 19:56:26 - INFO - root : ----------------------Data directory: /Users/hendrik/Projects/Thesis/YAIB/demo_data/mortality24/mimic_demo-----------------------
2023-08-19 19:56:26 - TUNE - root : Hyperparameter tuning disabled
2023-08-19 19:56:26 - TUNE - root : Choosing hyperparameters randomly from bounds.
2023-08-19 19:56:26 - INFO - root : Binding Hyperparameters:
2023-08-19 19:56:26 - INFO - root : LSTMNet.hidden_dim = 214
2023-08-19 19:56:26 - INFO - root : LSTMNet.layer_dim = 2
2023-08-19 19:56:26 - INFO - root : Adam.lr = 0.00018746261999642573
2023-08-19 19:56:26 - INFO - root : Training with these hyperparameters:
2023-08-19 19:56:26 - INFO - root : Binding Hyperparameters:
2023-08-19 19:56:26 - INFO - root : LSTMNet.hidden_dim = 214
2023-08-19 19:56:26 - INFO - root : LSTMNet.layer_dim = 2
2023-08-19 19:56:26 - INFO - root : Adam.lr = 0.00018746261999642573
2023-08-19 19:56:26 - INFO - root : ---------------Logging to /Users/hendrik/Projects/Thesis/yaib_logs/mimic_demo/Mortality24/LSTM/2023-08-19T19-56-26---------------
2023-08-19 19:56:26 - INFO - root : ========================================================STARTING TRAINING========================================================

2023-08-19 19:56:26 - INFO - lightning_fabric.utilities.seed : Global seed set to 2222
2023-08-19 19:56:26 - INFO - root : Using preprocessor: DefaultClassificationPreprocessor
2023-08-19 19:56:26 - INFO - root : Loading data from directory /Users/hendrik/Projects/Thesis/YAIB/demo_data/mortality24/mimic_demo
2023-08-19 19:56:26 - INFO - root : Generating splits.
Traceback (most recent call last):
  File "/opt/miniconda3/envs/yaib/bin/icu-benchmarks", line 33, in <module>
    sys.exit(load_entry_point('yaib', 'console_scripts', 'icu-benchmarks')())
  File "/Users/hendrik/Projects/Thesis/YAIB/icu_benchmarks/run.py", line 158, in main
    execute_repeated_cv(
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/Users/hendrik/Projects/Thesis/YAIB/icu_benchmarks/cross_validation.py", line 78, in execute_repeated_cv
    data = preprocess_data(
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/config.py", line 1605, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/utils.py", line 41, in augment_exception_message_and_reraise
    raise proxy.with_traceback(exception.__traceback__) from None
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/gin/config.py", line 1582, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "/Users/hendrik/Projects/Thesis/YAIB/icu_benchmarks/data/split_process_data.py", line 91, in preprocess_data
    data = make_single_split(
  File "/Users/hendrik/Projects/Thesis/YAIB/icu_benchmarks/data/split_process_data.py", line 172, in make_single_split
    dev, test = list(outer_cv.split(stays, labels))[repetition_index]
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/sklearn/model_selection/_split.py", line 342, in split
    X, y, groups = indexable(X, y, groups)
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/sklearn/utils/validation.py", line 443, in indexable
    check_consistent_length(*result)
  File "/opt/miniconda3/envs/yaib/lib/python3.10/site-packages/sklearn/utils/validation.py", line 397, in check_consistent_length
    raise ValueError(
ValueError: Found input variables with inconsistent numbers of samples: [1, 99]
  In call to configurable 'preprocess' (<function preprocess_data at 0x1600d52d0>)
  In call to configurable 'execute_repeated_cv' (<function execute_repeated_cv at 0x1472e5990>)
rvandewater commented 1 year ago

This bug should be fixed by #136. The debug flag might not work with the default cross-validation (5-fold) in classification mode, though, because the cross-validation will check for enough samples of each class to stratify the prediction.