Closed Nadav-Barak closed 2 years ago
Detailed well above, note that the display output for this depends on https://github.com/deepchecks/deepchecks/issues/1574 There are several algo challenges here therefore is recommended to review and approve with the team the part regarding methods to find relevant segments before moving on to the display calculations
A check that auto detects weak 2 dim segments for model based on desired score. Would replace as default both model error analysis and segment performance. check name TBD.
Implementation details: Calculate prediction error on test dataset. Train multiple tree, each based on only 2 features to predict user model prediction error. Trees should make sure that every leaf is of size 1%-10% of data Run several iterations with different hyper parameters on each 2 feature combination If takes a long time apply heuristics to minimize number of feature pairs. Save weakest leafs (segments) across all trees.
Visualization details: For each segment, create a heatmap such that the segment is visible + calculate other segments based on quantiles