SHAP with tensorflow only

Hello, I'm trying to visualize SHAP explainations of a tensorflow model. I'm using as reference the GradientExplainer, but I don't get how to use it on a tensorflow model (the example uses Keras). By reading "gradient.py" I understand that I need two tf.Operations to define my model. In the notebook model.layers[7].input and model.layers[-1].output are used, so I imagine that I have to extract from my tf graph an input layer I want to use and the corresponding output layer by using the get_operation_by_name function. What about the data, though? How can I define the correct form?

The data just needs to be in the same form that the input you specify expects. So in the example model.layers[0].input expects a 4D tensor of # samples x image width x image height x # channels. If you have lots and lots of data then a subsample of 1-10k samples should be plenty.

I've experimented a bit but I'm still stuck. Here are the details: I have a frozen model of an RFCN object detector trained in TensorFlow that I load with this function:

def worker():
    # Load a (frozen) Tensorflow model into memory.
    modelpath = 'path'
    detection_graph = tf.Graph()
    with detection_graph.as_default():
        od_graph_def = tf.GraphDef()
        with tf.gfile.GFile(modelpath, 'rb') as fid:
            serialized_graph = fid.read()
            od_graph_def.ParseFromString(serialized_graph)
            tf.import_graph_def(od_graph_def, name='')
    return detection_graph

With respect to the example shown on the notebook, the layer I want to explain is: LEV = model.get_tensor_by_name("SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0") which returns: <tf.Tensor 'SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0' shape=(?, ?, ?, 2048) dtype=float32>. This corresponds to model.layers[7].input in the example. It is clear that model.layers[0].input corresponds to IN = model.get_tensor_by_name('image_tensor:0') which returns <tf.Tensor 'image_tensor:0' shape=(?, ?, ?, 3) dtype=uint8>. The layer corresponding to the final layer (model.layers[-1].output in the example) is: OUT = model.get_tensor_by_name("SecondStagePostprocessor/Softmax:0") which returns: <tf.Tensor 'SecondStagePostprocessor/Softmax:0' shape=(?, ?) dtype=float32> The image to explain is loaded as numpy array into memory: data = np.expand_dims(to_explain, 0) and data.shape returns (1, 540, 960, 3)

I try to create the explainer calling e = shap.GradientExplainer((LEV, OUT), data) but I get

TypeError                                 Traceback (most recent call last)
<ipython-input-17-8437192e4bfc> in <module>()
      1 data = np.expand_dims(to_explain, 0)
----> 2 e = shap.GradientExplainer((LEV, OO), IN)

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in __init__(self, model, data, session, batch_size, local_smoothing)
     69 
     70         if framework == 'tensorflow':
---> 71             self.explainer = _TFGradientExplainer(model, data, session, batch_size, local_smoothing)
     72         elif framework == 'pytorch':
     73             self.explainer = _PyTorchGradientExplainer(model, data, batch_size, local_smoothing)

/home/optolab/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in __init__(self, model, data, session, batch_size, local_smoothing)
    182             self.gradients = [None]
    183         else:
--> 184             self.gradients = [None for i in range(self.model_output.shape[1])]
    185 
    186     def gradient(self, i):

TypeError: __int__ should return int object

I think the problem may be in the tensor dimensions which are unknown, while the original example loads the tensors with only the sample number unknown. So I've tried to get the tensors while the session is running but this way I get numpy arrays because I propagate the image to be explained into the model... In fact I use:

def infer(graph, image):
    with graph.as_default():
        with tf.Session() as sess:
            image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
            IN = tf.get_default_graph().get_tensor_by_name("SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0")
            OUT = tf.get_default_graph().get_tensor_by_name("SecondStagePostprocessor/Softmax:0")
            tensor_dict2 = {}
            tensor_dict2['IN'] = IN
            tensor_dict2['OUT'] = OUT
            new_dict = sess.run(tensor_dict2, feed_dict={image_tensor: np.expand_dims(image, 0)})
            return new_dict

And I get IN and OUT as arrays of shapes (1, 9, 16, 2048) and (100, 30) respectively. In this case if I try to create the explainer again with these arrays as inputs I can, in fact e = shap.GradientExplainer((IN, OUT), data) goes without errors... But I feel it's wrong.

To obtain shap values in the example it's used the map2layer function: shap_values, indexes = e.shap_values(map2layer(to_explain, 7), ranked_outputs=2) To my understanding, the map2layer functions basically calls the session and propagates the image in the desired layer. In fact the output is an array, I guess the same I obtain if I use my "infer" function... So I thought that maybe I can simply use the IN array obtained before: shap_values, indexes = e.shap_values(IN, ranked_outputs=2), but of course I get an error:

Traceback (most recent call last)
<ipython-input-22-3a98feed3725> in <module>()
----> 5 shap_values, indexes = e.shap_values(IN, ranked_outputs=2)
      6 index_names = np.vectorize(lambda x: class_names[str(x)][1])(indexes)

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    106         were chosen as "top".
    107         """
--> 108         return self.explainer.shap_values(X, nsamples, ranked_outputs, output_rank_order)
    109 
    110 

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    201 
    202         # rank and determine the model outputs that we will explain
--> 203         model_output_values = self.run(self.model_output, self.model_inputs, X)
    204         if ranked_outputs is not None and self.multi_output:
    205             if output_rank_order == "max":

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in run(self, out, model_inputs, X)
    290 
    291     def run(self, out, model_inputs, X):
--> 292         feed_dict = dict(zip(model_inputs, X))
    293         if self.keras_phase_placeholder is not None:
    294             feed_dict[self.keras_phase_placeholder] = 0

TypeError: unhashable type: 'numpy.ndarray'

At this point I think the problem is that the tensors I have to use when creating the explainer must be tensor objects not arrays, but I don't know how to get them correctly since if I just call them from the model I get unknown shapes and the explainer doesn't like them... If someone could lend me a hand it would be greatly appreciated!

I guess the problem is that Tensorflow models are not fully supported right now. Yes, I can pass a couple of tensors to SHAP but the tensors dimension problem is still there; Tensorflow graphs are more than just a sequence of layers like in Keras, so I guess the explainer cannot support them for now. I hope you guys will further extend SHAP in order to support TF graphs too!

The first error is because you have an unknown number of output classes (self.model_output.shape[1] is None), so GradientExplainer does not know how many outputs to try and explain. We could support that by evaluating the model and then assuming the number of output classes will remain fixed, but I am not sure we should make that assumption. Is there a good reason the number of output classes is not defined in the output tensor shape? If you really do have a fixed number of output classes could you try set_shape and see if that helps?

At a minimum is seems like a better error message would be good.

I tried as you suggested; I'll attach my code up to the error to better help you understand what I'm doing:

import numpy as np
import shap
import tensorflow as tf
from object_detection.utils import label_map_util
from PIL import Image
import codecs
def load_dependencies(): # qui ho due PATH
    labelspath = 'path/to/labelmap'
    nclasses = 29
    # Loading label map
    label_map = label_map_util.load_labelmap(labelspath)
    categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=nclasses, use_display_name=True)
    category_index = label_map_util.create_category_index(categories)
    return label_map, categories, category_index

def load_image_into_numpy_array(image): 
    (im_width, im_height) = image.size
    return np.array(image.getdata()).reshape((im_height, im_width, 3)).astype(np.uint8)

def worker():
    # Load a (frozen) Tensorflow model into memory.
    modelpath = 'path'
    detection_graph = tf.Graph()
    with detection_graph.as_default():
        od_graph_def = tf.GraphDef()
        with tf.gfile.GFile(modelpath, 'rb') as fid:
            serialized_graph = fid.read()
            od_graph_def.ParseFromString(serialized_graph)
            tf.import_graph_def(od_graph_def, name='')
    return detection_graph

def list_create(filename):
    # needed 'cause I load images path from a .txt file so I create a list of paths from there
    with codecs.open(filename, "r", encoding="utf-8-sig") as file:
        bbox = file.read().split('\n')
    arr = [str(r) for r in bbox]
    if arr[-1] == '':
        del arr[-1] # if I have an empty line at the end of file it gets deleted
    return arr

def infer(graph, image, flag):
    with graph.as_default():
        with tf.Session() as sess:
            image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
            IN = tf.get_default_graph().get_tensor_by_name("SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0")
            OUT = tf.get_default_graph().get_tensor_by_name("SecondStagePostprocessor/Softmax:0")
            tensor_dict = {}
            tensor_dict['IN'] = IN
            tensor_dict['OUT'] = OUT
            # Run inference
            if flag:
                # one image only
                new_dict = sess.run(tensor_dict, feed_dict={image_tensor: np.expand_dims(image, 0)})
            else:
               # background
                new_dict = sess.run(tensor_dict, feed_dict={image_tensor: image})
    return new_dict

lista = list_create('path/to/txtfile/containing/paths')
background = Image.open(lista[0])
background = load_image_into_numpy_array(background)
background = background.reshape(1,background.shape[0],background.shape[1],background.shape[2])

for i in range(1,len(lista)):
    I = Image.open(lista[i])
    I = load_image_into_numpy_array(I)
    I = I.reshape(1,I.shape[0],I.shape[1],I.shape[2])
    background = np.concatenate((background, I), axis=0)

to_explain = background[0]
label_map, categories, category_index = load_dependencies()
model = worker()
IN2 = model.get_tensor_by_name('image_tensor:0')
IN2.set_shape((None, 224, 224, 3))
LEV2 = model.get_tensor_by_name("SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0")
LEV2.set_shape((None, 16, 16, 2048))
OUT2 = model.get_tensor_by_name("SecondStagePostprocessor/Softmax:0")
OUT2.set_shape((None, 30))

new_dict = infer(model, background, 0)
only_image = infer(model, to_explain, 1)

e = shap.GradientExplainer((LEV2, OUT2), new_dict['IN'])
shap_values, indexes = e.shap_values(only_image['IN'], ranked_outputs=2)

By doing as you suggested I can go past the creation of e without errors, but when calling shap_values with the last line I now get:

RuntimeError                              Traceback (most recent call last)
<ipython-input-13-d3ae69c4e72a> in <module>()
      1 ###### explain how the input to the 7th layer of the model explains the top two classes
----> 2 shap_values, indexes = e.shap_values(only_image['IN'], ranked_outputs=2)
      3 index_names = np.vectorize(lambda x: class_names[str(x)][1])(indexes)

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    106         were chosen as "top".
    107         """
--> 108         return self.explainer.shap_values(X, nsamples, ranked_outputs, output_rank_order)
    109 
    110 

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    201 
    202         # rank and determine the model outputs that we will explain
--> 203         model_output_values = self.run(self.model_output, self.model_inputs, X)
    204         if ranked_outputs is not None and self.multi_output:
    205             if output_rank_order == "max":

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in run(self, out, model_inputs, X)
    293         if self.keras_phase_placeholder is not None:
    294             feed_dict[self.keras_phase_placeholder] = 0
--> 295         return self.session.run(out, feed_dict)
    296 
    297 

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    875     try:
    876       result = self._run(None, fetches, feed_dict, options_ptr,
--> 877                          run_metadata_ptr)
    878       if run_metadata:
    879         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1023       raise RuntimeError('Attempted to use a closed Session.')
   1024     if self.graph.version == 0:
-> 1025       raise RuntimeError('The Session graph is empty.  Add operations to the '
   1026                          'graph before calling run().')
   1027 

RuntimeError: The Session graph is empty.  Add operations to the graph before calling run().

To answer your question:

We could support that by evaluating the model and then assuming the number of output classes will remain fixed, but I am not sure we should make that assumption. Is there a good reason the number of output classes is not defined in the output tensor shape?

I'm using a pre-trained model obtained from TensorFlow Object Detection API Model Zoo; I guess the graph has a lot of "None" dimensions because the model is flexible according to the input image and the output classes defined by the user. If I'm not mistaken this way of using tensors with a lot of None dimensions is the dynamic instantiation, but I may be wrong!

It looks like SHAP is not using the right TensorFlow session. It tries to pick a good one by default, but you can also pass one with the session parameter of GradientExplainer. I would make sure you can call sess.run(...) yourself before calling SHAP, then pass session=sess to GradientExplainer.

Sorry for the late reply. I've tried this code:

modelpath = 'path/frozen_inference_graph.pb'
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(modelpath, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

    with tf.Session() as sess:
        image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
        IN = tf.get_default_graph().get_tensor_by_name("SecondStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/Relu:0")
        OUT = tf.get_default_graph().get_tensor_by_name("SecondStagePostprocessor/Softmax:0")
        OUT2 = OUT
        OUT2.set_shape((None,30))
        tensor_dict = {}
        tensor_dict['IN'] = IN
        tensor_dict['OUT'] = OUT
        # Run inference
        only_image = sess.run(tensor_dict, feed_dict={image_tensor: np.expand_dims(to_explain, 0)})
        new_dict = sess.run(tensor_dict, feed_dict={image_tensor: background})
        # SHAP
        e = shap.GradientExplainer((IN, OUT2), new_dict['IN'], session=sess)
        #{image_tensor: np.expand_dims(to_explain, 0)}
        shap_values, indexes = e.shap_values({image_tensor: np.expand_dims(to_explain, 0)}, ranked_outputs=2)

I'm a bit confused with the errors I get with e.shap_values. From the original tutorial, first the user calls "e" by using map2layer, mapping the whole background dataset of 50 images onto the layer to explain. This should corresponds to my calling of "new_dict", since the output of this is an array of values exactly as the output of map2layer. When calling e.shap_values in the original tutorial it's used map2layer again, this time mapping only the image to be explained onto the layer of choice. I guessed this should correspond to my only_image output but if I use it when calling shap_values, indexes = e.shap_values(only_image['IN'], ranked_outputs=2) I get:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-23-9b3c73915acc> in <module>()
     23         e = shap.GradientExplainer((IN, OUT2), new_dict['IN'], session=sess)
     24         #{image_tensor: np.expand_dims(to_explain, 0)}
---> 25         shap_values, indexes = e.shap_values(only_image['IN'], ranked_outputs=2)

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    106         were chosen as "top".
    107         """
--> 108         return self.explainer.shap_values(X, nsamples, ranked_outputs, output_rank_order)
    109 
    110 

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    201 
    202         # rank and determine the model outputs that we will explain
--> 203         model_output_values = self.run(self.model_output, self.model_inputs, X)
    204         if ranked_outputs is not None and self.multi_output:
    205             if output_rank_order == "max":

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in run(self, out, model_inputs, X)
    293         if self.keras_phase_placeholder is not None:
    294             feed_dict[self.keras_phase_placeholder] = 0
--> 295         return self.session.run(out, feed_dict)
    296 
    297 

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    875     try:
    876       result = self._run(None, fetches, feed_dict, options_ptr,
--> 877                          run_metadata_ptr)
    878       if run_metadata:
    879         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1098     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1099       results = self._do_run(handle, final_targets, final_fetches,
-> 1100                              feed_dict_tensor, options, run_metadata)
   1101     else:
   1102       results = []

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1270     if handle is None:
   1271       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1272                            run_metadata)
   1273     else:
   1274       return self._do_call(_prun_fn, handle, feeds, fetches)

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
   1289         except KeyError:
   1290           pass
-> 1291       raise type(e)(node_def, op, message)
   1292 
   1293   def _extend_graph(self):

InvalidArgumentError: You must feed a value for placeholder tensor 'image_tensor' with dtype uint8 and shape [?,?,?,3]
     [[Node: image_tensor = Placeholder[dtype=DT_UINT8, shape=[?,?,?,3], _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
     [[Node: map/while/LoopCond/_467 = _HostRecv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4108_map/while/LoopCond", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopmap/while/TensorArrayReadV3_1/_243)]]

Caused by op u'image_tensor', defined at:
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/user/.local/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 499, in start
    self.io_loop.start()
  File "/home/user/.local/lib/python2.7/site-packages/tornado/ioloop.py", line 1073, in start
    handler_func(fd_obj, events)
  File "/home/user/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 450, in _handle_events
    self._handle_recv()
  File "/home/user/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 480, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/user/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 432, in _run_callback
    callback(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/user/.local/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2714, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/user/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2818, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/user/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-23-9b3c73915acc>", line 8, in <module>
    tf.import_graph_def(od_graph_def, name='')
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3289, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3180, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/user/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

So I thought that maybe it wants the input tensor instead, but when calling shap_values, indexes = e.shap_values({image_tensor: np.expand_dims(to_explain, 0)}, ranked_outputs=2) I get another error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-24-b4d663bb6756> in <module>()
     23         e = shap.GradientExplainer((IN, OUT2), new_dict['IN'], session=sess)
     24         #{image_tensor: np.expand_dims(to_explain, 0)}
---> 25         shap_values, indexes = e.shap_values({image_tensor: np.expand_dims(to_explain, 0)}, ranked_outputs=2)

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    106         were chosen as "top".
    107         """
--> 108         return self.explainer.shap_values(X, nsamples, ranked_outputs, output_rank_order)
    109 
    110 

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in shap_values(self, X, nsamples, ranked_outputs, output_rank_order)
    201 
    202         # rank and determine the model outputs that we will explain
--> 203         model_output_values = self.run(self.model_output, self.model_inputs, X)
    204         if ranked_outputs is not None and self.multi_output:
    205             if output_rank_order == "max":

/home/user/.local/lib/python2.7/site-packages/shap/explainers/gradient.pyc in run(self, out, model_inputs, X)
    293         if self.keras_phase_placeholder is not None:
    294             feed_dict[self.keras_phase_placeholder] = 0
--> 295         return self.session.run(out, feed_dict)
    296 
    297 

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
    875     try:
    876       result = self._run(None, fetches, feed_dict, options_ptr,
--> 877                          run_metadata_ptr)
    878       if run_metadata:
    879         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/user/.local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1067             feed_handles[subfeed_t] = subfeed_val
   1068           else:
-> 1069             np_val = np.asarray(subfeed_val, dtype=subfeed_dtype)
   1070 
   1071           if (not is_tensor_handle_feed and

/home/user/.local/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
    490 
    491     """
--> 492     return array(a, dtype, copy=False, order=order)
    493 
    494 

TypeError: float() argument must be a string or a number

The error is clearly related to how the session.run is called, probably because in Tensorflow the feed_dict is defined differently... Do you think other Explainers will work in my case?

I am not sure without being able to run it what exactly the issue is. But the error is happening at: https://github.com/slundberg/shap/blob/5181fca8ab50bafc917fe9733fb6593ee73de357/shap/explainers/gradient.py#L203

which is simply trying to evaluate the model (nothing fancy). self.model_inputs and self.model_output are just IN and OUT2 for you and X is new_dict['IN']. If you have a chance I think it would be a reasonably simple task to try and run the model the same way yourself. self.run above is just:

https://github.com/slundberg/shap/blob/5181fca8ab50bafc917fe9733fb6593ee73de357/shap/explainers/gradient.py#L298-L302

shap / shap

SHAP with tensorflow only #515