search

MLD3 / FIDDLE-experiments

Experiments applying FIDDLE on MIMIC-III and eICU. https://doi.org/10.1093/jamia/ocaa139
24 stars 7 forks source link

Bugs in mimic3_experiments #2

Closed stdoo closed 3 years ago

stdoo commented 3 years ago

Hi Shengpu,

I have summarized some bugs in the mimic3_experiments directory. You may check them while available.

1_data_extraction

extract_data.py

Exceptions:

  1. Line 251: pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Result is too large for pandas.Timedelta. Convert inputs to datetime.datetime with 'Timestamp.to_pydatetime()' before subtracting.

Suggestions:

  1. Replace x.INTIME with x.INTIME.to_pydatetime().

LabelDistributions.ipynb

Exceptions:

  1. Line 44: FileNotFoundError
  2. Line 54: pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Result is too large for pandas.Timedelta. Convert inputs to datetime.datetime with 'Timestamp.to_pydatetime()' before subtracting.

Suggestions:

  1. Replace open('config.yaml') with open('../config.yaml')
  2. Replace x.INTIME with x.INTIME.to_pydatetime()

InclusionExclusion.ipynb

Exceptions:

  1. Line 29: pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Result is too large for pandas.Timedelta. Convert inputs to datetime.datetime with 'Timestamp.to_pydatetime()' before subtracting.

Suggestions:

  1. Replace x.INTIME with x.INTIME.to_pydatetime()

PopulationSummary.ipynb

Exceptions:

  1. Line 24: KeyError
  2. Line 26: FileNotFoundError
  3. Line 68: pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Result is too large for pandas.Timedelta. Convert inputs to datetime.datetime with 'Timestamp.to_pydatetime()' before subtracting.

Suggestions:

  1. Replace set_index('ICUSTAY_ID') with set_index('D')
  2. The file pop.mortality_benchmark.csv is not exist
  3. Replace x.INTIME with x.INTIME.to_pydatetime()

2_apply_FIDDLE

Suggestion: I think it's better to include FIDDLE module in this directory. After that, there are some other bugs.

README.md

Exceptions:

  1. Line 41: FileNotFoundError

Suggestion:

  1. There is no file named make_features.py

run_make_all.sh

exceptions:

  1. output_dir is required
  2. FileNotFoundError

Suggestion:

  1. You should set the output_dir for each run, since it's required in run.py

  2. Since the dir features/outcome=mortality,T=48.0,dt=1.0 is replaced by features/benckmark,outcome=mortality,T=48.0,dt=1.0 in 1_data_extraction/run_prepare_all.sh, this script is not able to run:

    OUTCOME=mortality
T=48.0
dt=1.0
python run.py \
    --data_fname="$DATAPATH/features/outcome=$OUTCOME,T=$T,dt=$dt/input_data.p" \

    Since the file pop.mortality_benchmark.csv is not exist, this script is not able to run:

    python run.py \
    --data_fname="$DATAPATH/features/benchmark,outcome=mortality,T=48.0,dt=1.0/input_data.p" \
    --population="$DATAPATH/population/pop.mortality_benchmark.csv" \

3_ML_models

lib/data.py

Exceptions:

  1. Line 75, 121: FileNotFoundError
  2. Line 123, 124: Directory not exist

Suggestion:

  1. The file pop.mortality_benchmark.csv is not exist
  2. The directory features/outcome=mortality,T=48.0,dt=1.0 is not exist and replaced by features/benckmark,outcome=mortality,T=48.0,dt=1.0

config.yaml

Exceptions:

  1. Line 21: The feature_dimension of ARF 4.0 is not 4143

Suggestion:

  1. Set to 4381

run_deep_eval.py

Exceptions:

  1. Line 57: import error

Suggestion:

  1. Replace from sklearn.externals.joblib import Parallel, delayed with from joblib import Parallel, delayed
daquang commented 3 years ago

Are there any plans to implement these fixes. I can confirm that these errors are still present in mimic3_experiments. Also, eicu_experiments has errors in the notebooks. The EICU notebooks make reference to icustays.csv, which is not in the EICU database, but is found in the MIMIC3 database.

shengpu-tang commented 3 years ago

Hello @daquang , yes I am currently working on fixing the issues in mimic3_experiments on a separate branch. Unfortunately due to package version differences, it might be impossible to reproduce the exact numerical results in the paper.

We are also working with physionet to share preprocessed datasets; more details will be available soon.

shengpu-tang commented 3 years ago

Also, eicu_experiments has errors in the notebooks. The EICU notebooks make reference to icustays.csv, which is not in the EICU database, but is found in the MIMIC3 database.

For eICU, the notebook generate_labels.ipynb creates the icustays.csv based on the patient table, and it contains the following columns:

Where partition specifies whether that ICU stay belongs to the train/val/test set. The split is done at the PatientID level so that for each patient, all their ICU stays always go into the same split.

I will add some usage instructions to eICU experiments as well.

shengpu-tang commented 3 years ago

Hello @stdoo @daquang, I'm excited to update you that preprocessed datasets for MIMIC-III and eICU are now available on physionet (https://physionet.org/content/mimic-eicu-fiddle-feature/1.0.0/)! I will clean up the code this weekend and close this issue after all bugs raised here have been addressed. Thanks again for your interest in our work.