search

OHDSI / DeepPatientLevelPrediction

An R package for performing patient level prediction using deep learning in an observational database in the OMOP Common Data Model.
https://ohdsi.github.io/DeepPatientLevelPrediction
10 stars 4 forks source link

All varImp is 0 #18

Open ChungsooKim opened 2 years ago

ChungsooKim commented 2 years ago

As @jreps mentioned prior meeting (21-10-04),

I also found that all variable importance of the fitted model is zero.

image

However, I also figured out the final model from the fitDeepNNTorch function still had values of predictors as weights and biases in each layer. so we need to discuss how to deal with these values for model reproducibility

model$parameters()

image

image

ChungsooKim commented 2 years ago

we can input model agnostic importances (interpretability) metrics like SHAP, LIME, or permutation importance in the VarImp column. but torch in R doesn't seem to have tools calculating those metrics yet.

chandryou commented 2 years ago

I thought about SHAP, but it would take tons of time to screen all covariates...

ChungsooKim commented 2 years ago

Yes, that's right. we will use millions of covariates from millions of patients. Then we can store all the original values of the weights for each layer in a separate column. However, if we prepare the code for calculating the importance metric in advance, users can only calculate the metric if they want (optionally). I would try to get agnostic codes for the torch.