Null explanation values

[baha-m]

i'm having issues with deepexplain being all 0 or -0 for some classes, i thought LRP is based on the taylor decomposition so there is no way that it could be all values 0 if the prediction value is not 0.

[marcoancona]

Does this only happens with e-LRP? All 0s might be caused by either all inputs at zero (independently on the output, as the output might be affected by biases) or by all gradients at zero (possibly also because of a problem in the network architecture or the way DeepExplain is used). I would need some more details to help.

[baha-m]

tryed this with e-LRP, it's not null inputs for sure and i don't think that i have a problem with gradients decay, this is the code i'm runing with the transformation needed:

`from sqlalchemy import create_engine import numpy as np from pandas import DataFrame from sklearn.preprocessing import LabelEncoder, StandardScaler import keras.models import pandas as pd

engine=create_engine('mysql://root:@localhost/claims') connection=engine.connect()

proxy=connection.execute('select * from valid')

proxy=connection.execute('select * from valid') df=DataFrame(proxy.fetchall()) df.columns =[i.upper() for i in proxy.keys()] df = df.replace('?', pd.np.nan).dropna(axis=0, how='any', subset=df.columns)

data=df rem=['INCIDENT_LOCATION','INSURED_ZIP','POLICY_ANNUAL_PREMIUM','POLICY_BIND_DATE','POLICY_NUMBER','INCIDENT_DATE','ID','FLAG']

rem=['INCIDENT_LOCATION','INSURED_ZIP','POLICY_ANNUAL_PREMIUM','POLICY_BIND_DATE','POLICY_NUMBER','INCIDENT_DATE','FRAUD_REPORTED']

for i in rem: del data[i]

cater sex

data['INSURED_SEX']=data['INSURED_SEX'].replace("MALE","1") data['INSURED_SEX']=data['INSURED_SEX'].replace("FEMALE","0") data['INSURED_SEX'].describe() data['INSURED_SEX']=data['INSURED_SEX'].astype(int)
cleanup={'POLICY_STATE':{'OH':2,'IL':1,'IN':0}, 'INCIDENT_STATE':{'OH':2,'NY':0,'SC':1,'WV':2,'VA':4,'NC':5,'PA':6}, 'AUTHORITIES_CONTACTED':{'Police':0,'Fire':1,'Other':2,'Ambulance':3,'None':4}, 'POLICY_CSL':{'100/300':0,'250/500':1,'500/1000':2}, 'INCIDENT_TYPE':{'Single Vehicle Collision':3,'Multi-vehicle Collision':2,'Parked Car':1,'Vehicle Theft':0}, 'INCIDENT_SEVERITY':{'Minor Damage':0,'Trivial Damage':1,'Major Damage':2,'Total Loss':3}, 'PROPERTY_DAMAGE':{'YES':1,'NO':0}, 'POLICE_REPORT_AVAILABLE':{'YES':1,'NO':0}} data.replace(cleanup,inplace=True) for i in cleanup.keys(): data[i]=data[i].astype(int) lb_make=LabelEncoder() label=['INSURED_EDUCATION_LEVEL','INSURED_OCCUPATION','INSURED_HOBBIES','INSURED_RELATIONSHIP','COLLISION_TYPE','INCIDENT_CITY','AUTO_MAKE','AUTO_MODEL'] for i in label: data[i]=lb_make.fit_transform(data[i]) sc=StandardScaler() relevent=list(data)

data=sc.fit_transform(data)

import numpy as np import keras.models from keras.models import model_from_json import tensorflow as tf

loading model

json_file = open('C:/Users/user/Desktop/demos/Release_version/model/model.json','r') model_json = json_file.read() json_file.close() model = model_from_json(model_json) model.load_weights("C:/Users/user/Desktop/demos/Release_version/model/model.h5") print("Loaded Model from disk") model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy']) graph = tf.get_default_graph()

ind is the value to change

ind=0 x=data[ind] with graph.as_default(): pred=model.predict(x.reshape(1,32)) print(pred)

from deepexplain.tensorflow import DeepExplain from keras import backend as K from keras.models import Model with DeepExplain(session=K.get_session()) as de: input_tensors=model.inputs fModel=Model(inputs=input_tensors, outputs=model.outputs) target_tensor=fModel(input_tensors) attributions=de.explain('elrp',target_tensor,input_tensors,x.reshape(1,1,32)) print('input: ',x) print('___') for layer in model.layers: print(layer.get_weights()[1]) print('****') print('___') print('Attributions: ',attributions)`

and for the execution you can visite this link to see the output: https://ibb.co/crMGPH i don't think it's a hight bias problem cause the null values i get them only for hight probability.

[marcoancona]

Does the model have multiple inputs? If not, the "input_tensor" param should be a single tensor, not a list. What about other methods (eg. grad*input), do you see the same phenomenon?

[baha-m]

same output with the other methods, you see a list of tensor in the code ? cause from where i stand the shape needs to be (1,32) 1 for a row and 32 for features.

[marcoancona]

I don't see the model from your comment above, but your screenshot show that the library outputs Model with multiple inputs: True. The call model.inputs returns a list, not a Tensor. Please see this example to pick a single tensor: https://github.com/marcoancona/DeepExplain/blob/master/examples/mint_cnn_keras.ipynb If the "normal" input to the network has shape (1, 32), the same shape should have the input when calling de.explain(), while you have (1, 1, 32) Still, I am not sure this is the problem. What kind of non-linear activation functions are used in the layers?

[baha-m]

relu for all layers and sigmoid for the output layer, and you are mistaken sorry mislead you with the first imput it was intended to show u that the input values are not null as for the input layer it's a tensor, model.inputs: [<tf.Tensor 'dense_1_input:0' shape=(?, 32) dtype=float32>]

[baha-m]

even after i updated the input calling for de.explain() in my code to have the shape of (1,32) still having the same results.

[marcoancona]

Ok, then I am afraid I would need to test myself to understand what is happening here. Can you provide a minimal functional code that reproduces the problem and that I can run?

[baha-m]

i have uploaded the model with minimal functional code that reproduces the same problem, i even added 2 other cases one of them it works and the other same problem: https://drive.google.com/file/d/1sWWjZcb6onhnTVriDUfN726C5z9wUqwZ/view?usp=sharing

[marcoancona]

Thanks, I managed to reproduce it. However, I found that is the expected behavior. If your print the output of the second last layer with the following code

    fModel=Model(inputs=input_tensors, outputs=model.layers[2].output)
    print(fModel.predict(x.reshape(1,32)))

you can see that all outputs are 0, meaning that all ReLUs are not activated (with zero gradient) and no gradient-based method attribute credits to the input. The "probability" in output is only due to the bias. Since you have only a small number of features and therefore performance is not an issue, I would suggest to use the occlusion method instead.

[baha-m]

how did i miss that, i'm so sorry for the trouble and thank you allot for bearing with me and thank you for the advice it makes more sens now that i read the occlusion description.