Lightwood is an AutoML library that aims at abstracting common procedures in the Machine Learning workflow (e.g. data preparation, feature engineering, model building, model analysis) with an approach that can be fully automated but easily customizable and extended thanks to its flexibility.
To analyze a model, a series of "analysis blocks" run after the ensemble step and when generating predictions. Each of these blocks can (1) do some type of analysis after training has finished, (2) do computations at inference time to give additional insights about predictions (e.g. a confidence score), or both (1) and (2).
In this task, we seek to add an analysis block that will output anchor explanations for the predictions of Lightwood models.
Steps :male_detective: :female_detective:
Fork the Lightwood repository, checkout the staging branch and from it create a new one.
Add an analysis block (inherit from lightwood/analysis/base/BaseAnalysisBlock) in lightwood/analysis/helpers/ that computes anchors in its .explain() method by interacting with the library linked above. The output should be added to the input insights in additional columns with a naming scheme like f"prediction_anchor", or similar, with a string containing the entire rule (e.g. "COLUMN_NAME >= VALUE").
Create (or modify an existing) integration test that uses your analysis block and checks the output has the expected properties. Alternatively, you can create a tutorial in the docssrc/source/tutorial/custom_explainer folder, explaining how to use this block and what insights can be made from an example predictive task.
Make the PR and address any comments that reviewers might make.
paxcema
commented
5 months ago
Introduction
Lightwood is an AutoML library that aims at abstracting common procedures in the Machine Learning workflow (e.g. data preparation, feature engineering, model building, model analysis) with an approach that can be fully automated but easily customizable and extended thanks to its flexibility.
To analyze a model, a series of "analysis blocks" run after the ensemble step and when generating predictions. Each of these blocks can (1) do some type of analysis after training has finished, (2) do computations at inference time to give additional insights about predictions (e.g. a confidence score), or both (1) and (2).
Anchors (High-Precision Model-Agnostic Explanations) are a popular method for explaining predictions of a ML model for tabular data.
Task
In this task, we seek to add an analysis block that will output anchor explanations for the predictions of Lightwood models.
Steps :male_detective: :female_detective:
stagingbranch and from it create a new one.lightwood/analysis/base/BaseAnalysisBlock) inlightwood/analysis/helpers/that computes anchors in its.explain()method by interacting with the library linked above. The output should be added to the input insights in additional columns with a naming scheme likef"prediction_anchor", or similar, with a string containing the entire rule (e.g. "COLUMN_NAME >= VALUE").docssrc/source/tutorial/custom_explainerfolder, explaining how to use this block and what insights can be made from an example predictive task.Additional rewards :1st_place_medal:
Accepted PRs will let you participate in the MindsDB hacktoberfest contest! For more info check out https://mindsdb.com/hacktoberfest/.