Can't use explainer with Keras neural network

[heytalt]

Hi,

I am trying to find which part of my signal is important in my NN. My data consist of simple sequences of length 1488 and I need to classify it between 20 classes. The Model I use is the following:

model = tf.keras.Sequential([
     tf.keras.layers.Flatten(input_shape=(x.shape[1],1)),  # input layer (1)
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(250, activation='relu'),  
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(125, activation='relu'),  
    tf.keras. layers.Dense(len(df["icao24"].unique()), activation='softmax')
])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

I already tried to run your exemple notebook and it is working, but when I try to run the following code after having fitted my Model:

num_features = 1 # how many feature contained in explanation num_slices = 48 # split time series series = X_test[0]

explainer = LimeTimeSeriesExplainer(class_names=['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16', '17','18','19','20']) exp = explainer.explain_instance(series.reshape(1,-1), model.predict, num_features=num_features, num_samples=3, num_slices=num_slices, replacement_method='total_mean')

I get the following error code:

1 explainer = LimeTimeSeriesExplainer(class_names=['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16', 2 '17','18','19','20']) ----> 3 exp = explainer.explain_instance(series, model.predict, num_features=num_features, num_samples=4, num_slices=num_slices, replacement_method='total_mean')

... HUGE ERROR CODE THAT I THINK IS IRRELEVANT

ValueError: in user code:

ValueError: Layer sequential_5 expects 1 inputs, but it received 4 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:2' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:3' shape=(None, 1488) dtype=float32>]

Do you know what is wrong or does Lime_time series package does not support keras models ?

I would appreciate some help a lot!

[emanuel-metzenthin]

Hi,

Sorry for answering so late. It seems to me (without having done too much digging) that the keras model.predict function only accepts one sample. At least that is what the error says. As you set num_samples in LIME to 3 it will pass a batch of 3 sample series to the model.predict function. Make sure to use a function that supports batches.

Best regards! On 30. Oct 2020, 15:39 +0100, hey_talt notifications@github.com, wrote:

Hi,

I am trying to find which part of my signal is important in my NN. My data consist of simple sequences of length 1488 and I need to classify it between 20 classes. The Model I use is the following:

model = tf.keras.Sequential([ tf.keras.layers.Flatten(input_shape=(x.shape[1],1)), # input layer (1) tf.keras.layers.BatchNormalization(), tf.keras.layers.Dense(250, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(125, activation='relu'), tf.keras. layers.Dense(len(df["icao24"].unique()), activation='softmax') ])

model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

I already tried to run your exemple notebook and it is working, but when I try to run the following code after having fitted my Model:

num_features = 1 # how many feature contained in explanation num_slices = 48 # split time series series = X_test[0]

explainer = LimeTimeSeriesExplainer(class_names=['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16', '17','18','19','20']) exp = explainer.explain_instance(series.reshape(1,-1), model.predict, num_features=num_features, num_samples=3, num_slices=num_slices, replacement_method='total_mean')

I get the following error code:

1 explainer = LimeTimeSeriesExplainer(class_names=['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16', 2 '17','18','19','20']) ----> 3 exp = explainer.explain_instance(series, model.predict, num_features=num_features, num_samples=4, num_slices=num_slices, replacement_method='total_mean')

... HUGE ERROR CODE THAT I THINK IS IRRELEVANT

ValueError: in user code:

ValueError: Layer sequential_5 expects 1 inputs, but it received 4 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:2' shape=(None, 1488) dtype=float32>, <tf.Tensor 'IteratorGetNext:3' shape=(None, 1488) dtype=float32>]

Do you know what is wrong or does Lime_time series package does not support keras models ?

I would appreciate some help a lot!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/emanuel-metzenthin/Lime-For-Time/issues/6, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEOWKJGTU637L6XWC2CZWZ3SNLFZBANCNFSM4TFBTGLQ.

[heytalt]

Hi,

I think my model is able to predict batches.

For exemple model.predict(X_test[:5]) works.

It is also possible to do predictions using model(X_test[:5],training=False) .

So when I tried to put model(training=False) instead of model.predict as argument inside of explainer.explain_instance I got another error code:

ValueError: The first argument to Layer.call must always be passed.

Which I don't think is really useful unfortunately. Just in case you want to reproduce my errors, I send you my "whole" code:

# x and y are numpy arrays. x has shape(166410, 1488).
X_train, X_test, y_train, y_test = train_test_split(x, y, shuffle=True, 
                                                    train_size=0.67)
class_weights = dict(zip(np.unique(y_train), class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)))

model = tf.keras.Sequential([
     tf.keras.layers.Flatten(input_shape=(x.shape[1],1)),  # input layer (1)
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Dense(250, activation='relu'),  # hidden layer (2)
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(125, activation='relu'),  # hidden layer (2)
    tf.keras. layers.Dense(len(df["icao24"].unique()), activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(X_train,y_train,epochs=30, batch_size=128,class_weight=class_weights)

num_features = 1 # how many feature contained in explanation
num_slices = 48 # split time series
series = X_test[0]

explainer = LimeTimeSeriesExplainer(class_names=['0','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16',
                                                 '17','18','19','20'])
exp = explainer.explain_instance(series, model(training=False), num_features=num_features, num_samples=4, num_slices=num_slices, replacement_method='total_mean')
exp.as_pyplot_figure()