[FEAT] Explainability Checks - research and characterize

deepchecks / deepchecks

Deepchecks: Tests for Continuous Validation of ML Models & Data. Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling to thoroughly test your data and models from research to production.

https://docs.deepchecks.com/stable

Other

3.56k stars 249 forks source link

[FEAT] Explainability Checks - research and characterize #1716

Open shir22 opened 2 years ago

shir22 commented 2 years ago

Explore existing explainability checks.
Differentiate between "exploratory" and "testing" explainability issues.

DoD: Present overview of top relevant directions (based on existing research, trends and solutions). Suggest 3-5 explainability checks for the package.

nirhutnik commented 2 years ago

Tasks:

Search for sources - 2 SPs
Create list of possible candidates and expand by algo brainstorm - 1 SP
Prioritize list and show top candidates to Shir + Yaron

noamzbr commented 2 years ago

Elaboration on task: Suggested checks should be checks that use explainability to test some property of the model or data. They should either have a clear condition on the "OK" and "Not OK" situation, or at least have a display that clearly indicates if the current state is "OK".

Example for potentially good idea - use the diff of some FI metric between train and test to find drift Example for not-so-good idea - display SHAP beeswarm plot as is