search

Trusted-AI / AIX360

Interpretability and explainability of data and machine learning models
https://aix360.res.ibm.com/
Apache License 2.0
1.61k stars 305 forks source link

CEMExplainer support for tabular data with categorical features #93

Open rahulnair23 opened 4 years ago

rahulnair23 commented 4 years ago

I'm looking to use the contrastive explainer on tabular data which the docs state is supported.

What is the recommended mechanism to deal with categorical features for this explainer?

I've one-hot encoded and then normalized like so:

c_transformer = Pipeline(steps=[('onehot', OneHotEncoder(handle_unknown='ignore')),
                                ('functr', FunctionTransformer(lambda x: x.toarray(), accept_sparse=True)),
                                ('scalar', MinMaxScaler(feature_range=(-0.5, 0.5)))])

The resulting pertinent negatives and positives adjust all values of a category. As an example, here is the delta_pn (which I understand to be the difference needed change the classification) for the sex feature which is binary in this dataset.

sex_Female                                   0.500000
sex_Male                                    -0.500000

The change impacts both categories. Its unclear how to do the inverse transform for these cases when using one-hot encoding.

sadhamanus commented 4 years ago

The CEM explainer will treat all values as real. For special treatment of categorical variables as well as only black-box access you will have to look at its model agnostic version (MACEM) which is not part of the toolkit but is available through an IBM product called Watson Openscale. Hope this helps.