Output values are outside the image

[meet10may]

I am using the innvestigate package to visualize my deep learning model and I have encountered a strange situation, where the innvestigate methods compute values that are outside the image! I have tried with deep_taylor, integrated gradients, gradient, and all seem to compute values that lie outside the image. There is no signal outside the image(the brain)!

However, if I compute the Shapley values from shap package, the values are reasonable, and they are all inside the brain(which makes sense).

The model is created using tensorflow1.12. I have attached the screenshots. The input training data is task-based fMRI data of size 9032x91x109x91x1. Additionally, there is no issue with the trained model as it performs the task as expected!

Here is a piece of code:

import innvestigate
from keras.models import load_model
from nilearn.plotting import view_img

model=load_model(model_to_use)
method = 'gradient'
analyzer = innvestigate.create_analyzer(method, model)
first_trial_analyser = analyzer.analyze(test_imgs[0:1,:]) # test_imgs has 2916x91x109x91x1 dimension, so visualizing first trial.
# Converted the first_trial_analyser to nifti format
...
...
view = view_img(first_trial_analyser)

Could you please provide your thoughts on it and where I am doing wrong?

Thank you so much!

Result using Innvestigate Result using Shap

[sebastian-lapuschkin]

Hi @meet10may,

can you verify whether the background areas contain only zero-valued inputs when fed into the network (after preprocessing the inputs)? If already gradient computation leads to non-zero attribution in the background area, I would expect some neurons (ie inputs) firing in these locations. It would be interesting to see the lowest layers outputs corresponding to in these areas.

[adrhill]

I'm closing this issue due to inactivity. Feel free to reopen if the problem still exists.