BUG: Sum of Shapley values does not equal difference between prediction and expectation value in keras models

karllandheer commented 10 months ago

Issue Description

Hello, I am having an issue where the sum of the shap values does not sum to the model prediction - expected_value. The model is in keras. I have included a minimum reproducible example. I tried both regression (i.e., linear last layer), and classification (sigmoid). I believe in both cases this equality should hold, however both were not for me.

Minimal Reproducible Example

import os
import zipfile
import numpy as np
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers

import shap

from tensorflow.compat.v1.keras.backend import get_session
tf.compat.v1.disable_v2_behavior()

def get_model(width=128, height=128, depth=64):
    """Build a 3D convolutional neural network model."""

    inputs = keras.Input((width, height, depth, 1))

    x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(inputs)
    x = layers.MaxPool3D(pool_size=2)(x)
    x = layers.BatchNormalization()(x)

    x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(x)
    x = layers.MaxPool3D(pool_size=2)(x)
    x = layers.BatchNormalization()(x)

    x = layers.Conv3D(filters=128, kernel_size=3, activation="relu")(x)
    x = layers.MaxPool3D(pool_size=2)(x)
    x = layers.BatchNormalization()(x)

    x = layers.Conv3D(filters=256, kernel_size=3, activation="relu")(x)
    x = layers.MaxPool3D(pool_size=2)(x)
    x = layers.BatchNormalization()(x)

    x = layers.GlobalAveragePooling3D()(x)
    x = layers.Dense(units=512, activation="relu")(x)
    x = layers.Dropout(0.3)(x)

    #outputs = layers.Dense(units=1, activation="sigmoid")(x)
    outputs = layers.Dense(units=1, activation="linear")(x)

    # Define the model.
    model = keras.Model(inputs, outputs, name="3dcnn")
    return model

# Build model.
width = 64
height = 64
depth = 64

model = get_model(width=width, height=height, depth=depth)
model.summary()

data = np.random.random((5,width,height,depth,1))
explainer = shap.DeepExplainer(model, data[0:4,:,:,:,:])

preds_raw = model.predict(data[4:,:,:,:,:])

difference = preds_raw-explainer.expected_value
shap_values = np.squeeze(explainer.shap_values(data[4:,:,:,:,:]))
assert np.isclose(difference, np.sum(shap_values)), "keras shapley does not sum up correctly"

Traceback

assert np.isclose(preds_raw, np.sum(shap_values)), "keras shapley does not sum up correctly"

Expected Behavior

No response

Bug report checklist

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest release of shap.
[ ] I have confirmed this bug exists on the master branch of shap.
[ ] I'd be interested in making a PR to fix this bug

Installed Versions

shap==0.42.1 tensorflow==2.13.0 numpy==1.24.3

karllandheer commented 10 months ago

I have found that this is because difference-np.sum(shap_values) is typically <0.01, which is the threshold used in deep_tf.py, hence why it's not caught automatically by the check_additivity=True. I am not super familiar with SHAP, is the equality not really an equality but subject to some approximations? (i.e., maybe there's no bug, i'm just being accidentally pedantic)

CloseChoice commented 9 months ago

The tolerance that is acceptable is 0.01, so your example lies within that. See here

CloseChoice commented 7 months ago

@karllandheer any objections to closing this issue?

shap / shap