PFI, LOCO and Correlated Features

[ThomasAlt]

To Do:

• better simulation of correlated and uncorrelated features (make use of short cuts ect.)
• apply all scenarios on LOCO and PFI
• make code shorter, the main focus was that everything works
• Simulation with real data is missing

Notes:

• The code of LOCO Feature Importance takes some time (30-40 sec)
• Please review my implementation of LOCO feature importance

[veronikakro]

Simulations:

• add paragraph on how data was simulated, i.e. how the settings differ from each other and what they have in common
• establish a structure to outline what exactly you are investigating; currently it looks like you will end up in a large number of simulations and plots due to many different dimensions of interest, such as: o degree of correlation: six options planned --> would three options be enough to underline your findings (e.g. 0.99 vs. 0.6 vs. 0)? o data generating process: linear, nonlinear dependence, etc. o algorithms: random forest vs. linear model o number of feature variables
• add some more explanation, e.g. why does random forest come to completely different conclusions in respect of feature importance compared to the linear model?

Plots:

• add headlines and captions
• make use of ggarrange() or similar functions to visually compare the two models (left vs. right) and different correlations (top vs. bottom) at the same time
• align the x-axis
• LOCO-Plots: highlight dots in eye-catching colours instead of white

Other:

• Some spelling errors to be fixed (e.g. ‚scenario‘ instead of ‚szenario‘)
• Set seed when modelling with resampling etc. to make results reproducible

[veronikakro]

LOCO

• Your comment "Problem to order the importance within the graphic.": This issue should be fixed by: ploco0 <- ggplot(data = FIP0 ,aes(reorder(x=feature02, X=imp0), y = imp0)) + #### hier fehler geom_point(size=3, shape=21, fill="white") +... (delete aes from geom_point and reorder x-values according to their importance)

• loop can be simplified: result0 <- data.frame(matrix(nrow=1,ncol=length(feat0))) for(i in 1:length(feat0)){ taskfeat0 = dropFeatures(task0, feat0[i]) mod.feat0 = train(learner0, taskfeat0) resfeat0 = resample(learner = learner0, task = taskfeat0, resampling = res_desc,show.info = FALSE); result0[i] = resfeat0$aggr-res$aggr }

[ThomasAlt]

There is a unfortunate/wrong sentence in the summary: "In the literature other learning algorithms also showed similar results, like the random forest calculated by the out-of bag observations or neural networks." Of course, the random forest is not another learning algorithm. Here the Feature Importance is calculated using the OOB observations. This sentence should be corrected before publishing the book.