Regression diagnosis notes

[ianozsvald]

• Add Gradient Boosted Tree Partial Dependence plot - does it agree with my 2 variable exploration with RF?
• Add more cumulative plots
• Reorder so the experimental stuff is at the end, the start is about explaining predictions
• Quantile regression intervals (on RF or GBT?) using fully expanded trees
  • http://blog.datadive.net/prediction-intervals-for-random-forests/
• Using LIME and a set of test cases show which features are most important over a regression range - do we see a pattern?
• Consider adding Shapely
• Confusion matrix (used binned data) ?
• Can I extract LIME features to explain what's in the model?
• Consider Breiman's RF suggestion of randomizing each key feature in turn in the test set, then observing the overall change in the score - which variables are critical? How well does this mirror feature-importance? -> recently added to ELI5 https://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html