MIMIC - Evaluate Drug Forward Prediction

[dancoster]

• RMSE/nRMSE metrics
• Boxplots with AUPR/AUC

[dancoster]

nRMSE plot - per each lab measurement (e.g. Glocose): X-axis - signficant drugs (with low p-value of drug lab pair) Y-Axis - nRMSE Per each drug, generate bar per each imputation methods (ffill, median, mean, drug forward).

do the same for RMSE

[rfeinsod]

[rfeinsod]

I must admit that they look kind of off, what do you think?

[dancoster]

Agree, which drug<>labs pairs were used for the imputation?

[rfeinsod]

I finally got the aupr/auc to run well

[dancoster]

Cook!

[dancoster]

Cool*

[dancoster]

• Input characteristics - number of samples, number of targets (how many died), 5-fold cross validation
• Which pairs were used for imputation
• On which patients the AUPR / AUROC / RMSE / nRMSE is evaluated

[rfeinsod]

Pairs That were used are attached

relevant_pairs.csv

Input characteristics: These runs were done on 1000 patients. In the specific subset that the above figures were generated for - target was False for all. I didn't conduct 5-fold cross validation here

The tests were conducted on 100 patients from the output of the function new_load_data.

[dancoster]

(1) I'm pretty sure that the imputation is not working on more than one MED per Lab. Can you please double check if that happens? (2) Please re run the function with 10,000 patients (obviously no model can learn were all the target are from the same class, i.e., False) (3) I'm pretty sure that 5-crossfold validation was conducted, because the box plots represent the AUROC/AUPR on each fold (of the 5 cross-fold validation). (4) What do you mean by 'The tests were conducted on 100 patients from the output of the function new_load_data.'?

[rfeinsod]

1) Uploaded the RMSE results from last week - Note that results are generated for several labs per med

df_results_rmse_2024-08-16_14-40-45.csv

2) Rerunning - will probably run all night 3) I'll check it out 4) I meant that while there was some processing done, no additional filtering was conducted. Meaning this was run on a subset of all patients

[rfeinsod]

2) Attached are additional graphs from re-running the RMSE/nRMSE on 10k patients with a min patient value of 1k. Please note that this was run on a smaller selection of pairs (because of the higher min patient value).

[rfeinsod]

3) My mistake- kflod validation as done as part auc/aupr generation.

[dancoster]

[1] My question was related to the AUPR/AURPC calculation (were more than one type of drug can affect a lab measurement, that's relate to the relevant pairs.csv ). 'df_results_rmse_2024-08-16_14-40-45.csv' is related to RMSE/nRMSE calculation. Check how lab measurement is being imputed when more than one drug has been given. and run rsults AUPR/AUROC. [3] Let's disscuss -done [6] Create RMSE plot:

• fix all (in df_results_rmse_2024-08-16_14-40-45.csv)
• create separated bar plots for drugs, subject_indices
• CR for drugs_indices, subject_indice Subject indices: Calculate RMSE only on subjects that got the drug anytime during their hospitalization. drugs indices: Calculate RMSE only on indices that were affected by drug_forward imputation all: all the subjects [7] Validate how many patients were used to calc AUROC/AUPR

[dancoster]

• add as a input param to the function, the way calculating rmse/nRMSE = [indices, subjects, all]
• Later on - manual checking different 'after' time windows and their effect on RMSE.

[rfeinsod]

RMSE can be run succesfully on any variable. For Example on all here

[dancoster]

[1] Check how lab measurement is being imputed when more than one drug has been given. Than clarify which pairs were used when AUPR/AUROC run.