deepchecks / deepchecks

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.
https://docs.deepchecks.com/stable
Other
3.6k stars 254 forks source link

[FEAT] [ModelEvaluation] Model performance drill down check #1571

Closed Nadav-Barak closed 2 years ago

Nadav-Barak commented 2 years ago

A check that auto detects weak 2 dim segments for model based on desired score. Would replace as default both model error analysis and segment performance. check name TBD.

Implementation details: Calculate prediction error on test dataset. Train multiple tree, each based on only 2 features to predict user model prediction error. Trees should make sure that every leaf is of size 1%-10% of data Run several iterations with different hyper parameters on each 2 feature combination If takes a long time apply heuristics to minimize number of feature pairs. Save weakest leafs (segments) across all trees.

Visualization details: For each segment, create a heatmap such that the segment is visible + calculate other segments based on quantiles

Screen Shot 2022-06-02 at 15 10 14

Nadav-Barak commented 2 years ago

Detailed well above, note that the display output for this depends on https://github.com/deepchecks/deepchecks/issues/1574 There are several algo challenges here therefore is recommended to review and approve with the team the part regarding methods to find relevant segments before moving on to the display calculations