using KNN instead of perturbing to produce neighborhood?

[ihadanny]

Would it be useful to produce the neighborhood from the training set with KNN instead of generating neighbors by perturbing? It might be handy when there is a big training-set to choose neighbors from, and there are relationships between the features that might be broken when perturbing every feature by itself - thus creating an 'illegal' neighbor

[marcotcr]

Not from the training set, as that is what is used to train the model (and thus predictions on it are not a good measure of how the model predicts elsewhere). Maybe from a validation set, but it's hard to imagine a problem where you would have enough data set apart for validation for this to make sense.

[ihadanny]

Agree - from the validation set. Suppose I have a big validation set, do you think its better to use KNN or simply random sample from it? As the notion of locality is already "handled" by the weights of the explainer model...

[ihadanny]

come to think of it - why is it wrong to use the training data for generating the neighborhood?

The current method of perturbation uses the distributions of the training data for generating the neighborhood. You could have claimed the same claim: "predictions on it are not a good measure of how the model predicts elsewhere" on that approach, and demand that the user provide the validation data for estimating the mean and scale of the features...

[marcotcr]

In answer to the first question: with a big validation set, I think simply sampling from it is a good idea for tabular data. As you mention, I think locality is already covered by the weighting step. For text, data is so sparse that the validation set would need to be enormous.

About using the training data: the current method of perturbation assumes that features are independent and samples according to training set proportions (in the case of categorical), or according to a normal (in the case of continuous). We would expect the validation set to have similar statistics, but it is a different thing to actually use the instances used for training. Think of a classifier that completely overfits by creating a table where each individual example in the training set is assigned to its class (i.e. exact matching of training instances), and I think the reason will become clear.

[ihadanny]

Ok, thanks a lot for these answers! I think that the main thing I'm trying to understand is this:

Say I have a sample which is an outlier. My black-box classifier uses an esoteric combination of features to classify it, discerning it from its neighbors.

Now the current explainer perturbs around the center of mass of the training set, and then fits a distance-weighted simple model.

Wouldn't it be better if I would perturb around the explained sample itself and then fit a distance-weighted simple model?

It's just that in a sparse world, I'm worried that the explainer would not understand that given a set of features (e.g. age>70, bilirubin < 1.5 and BMI > 3.0) it is essential to classify using a usually-non-important feature (e.g. hemoglobin level)

Did you try that approach? I guess it can be easy to change and perturb around the sample and check if its indeed giving better explanations, but I think that there's something that I'm missing here since you didn't mention that approach at all :(

[marcotcr]

Notice that perturbing around the explained sample is not the same as using the k-nearest neighbors. I did try this - sampling from a normal centered at the feature value for the current instance (in the case of continuous attributes).

The problem you outline with outliers is a real one, but sampling around the explained sample has its own problems - imagine the explained sample has a really extreme value (e.g. age > 98). We may not see any instances with age < 60 if we sample around that point. If the classifier has a fixed threshold on age at 55, we will think that age is not an important factor (since perturbing it does not change the prediction).

Honestly, I don't know which is better. I thought perturbing around the sample would be less robust with a fixed width (i.e. there's an extra parameter to chose, how 'far' away we're sampling).

[christophM]

Maybe you could sample both from the center of the training data AND sample around the the instance of interest? Then combine the two samples and proceed as usual with the locally weighted kernel

[ihadanny]

Nice idea @christophM ! but again I stumble upon the problem of evaluating which method is better... I read the LIME paper, but I really don't have a feel how can I assess which approach is better quantitatively, so right now I'm just trying different flavors of neighborhood selection and looking at the features they selected and deciding if they make sense to me...

[christophM]

It will depend on the data and features used, I guess.

In the end you would like to evaluate which explanation is better, but as far as I know there is no good metric for that.