KerasClassifier model weights does not look properly reset before being fit

[nabilalibou]

Hello,

I still has some bug/incomprehension with the Scikeras and Sklearn fit function interaction. Last issue was done in a hurry and did not highlight the bug so here I am again. So I have the impression that fitting a model wrapped in KerasClassifier in a loop will not reset its weights before fitting it to the data in the same manner as If the Keras model alone is instantiated in the loop before being fit.

In summary, It seems that the model is not correctly reset so fitting it in a loop (a KFold cross-validation loop). I encountered this issue using Scikeras professionally (there, I had clearly models that gave me an accuracy of 0.6 in the first loop turn before skyrocketing to something close to 1. I tried to reproduce the behavior with standard dummy data but performances depends too much on the data quality so I wanted to highlight the issue by looking directly at the weight of biases (even if it less intuitive to me than sudden performance increase).

Here the 2 piece of codes using a dummy dataset and printing the weight of the model's bias:

Reinstantiate the model directly in the loop:

from numpy import loadtxt
import numpy as np
from scikeras.wrappers import KerasClassifier
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import accuracy_score
from sklearn.utils import shuffle
from sklearn.pipeline import make_pipeline

def model_nn():

    model = Sequential()
    model.add(Dense(12, input_shape=(8,), activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['binary_crossentropy'])
    return model

# load the dataset
dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
model_nn = model_nn()

n_split = 10
X, y = dataset[:, 0:8],  dataset[:, 8].astype(int)
num_runs = 30
all_scores = []
kf = KFold(n_splits=n_split, shuffle=True)
for train_index, eval_index in kf.split(X, y):
    x_test, x_train = X[eval_index, :], X[train_index, :]
    y_test, y_train = y[eval_index], y[train_index]
    scores = []
    score = 0
    for nrun in range(num_runs):
        x_, y_ = shuffle(x_train, y_train)
        biais = model.layers[1].get_weights()[1]
        print(f"Before: {biais}")
        model_nn.fit(x_, y_, epochs=30, batch_size=16, verbose=False)
        biais = model.layers[1].get_weights()[1]
        print(f"After: {biais}")
        y_pred = (model_nn.predict(x_test) > 0.5).astype(int)
        score += np.mean(accuracy_score(y_test, y_pred))
    score /= num_runs
    all_scores.append(score)
print(f"{Mean accuracy: np.mean(all_scores)}")

Some of the logs showing the weights:

Console:

Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.07789122  0.44471914  0.17739178  0.18815784 -0.06638421 -0.134316
-0.19207452  0.        ]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.19981909 -0.11162576  0.27216756 -0.26187867  0.2923611  -0.22338101
0.31982166 -0.12609349]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [ 0.         -0.17140345 -0.12318961  0.28291494  0.26646045 -0.15588419
-0.01235106 -0.12218504]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.05344929 -0.05919147 -0.2310531   0.04890905  0.42504674 -0.04857899
-0.11618819  0.04065897]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [ 0.1234226  -0.15809119  0.17418759  0.          0.         -0.01723221
-0.02341213 -0.2070783 ]
Mean accuracy: 0.6363636363636364
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [ 0.          0.09611724  0.36199677 -0.03512564 -0.2497901  -0.14980865
-0.2800358  -0.20440115]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [ 0.          0.          0.          0.1239132  -0.08133116 -0.0666899
-0.12992416  0.        ]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.05295666 -0.13505259  0.05375762  0.1768484  -0.07555941 -0.00511672
-0.09020937  0.00096915]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [ 0.19113357 -0.25497457 -0.17226881  0.04259212 -0.27500728  0.
-0.08212745  0.26095137]
Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.02211879 -0.2620719  -0.27976933  0.3668856   0.3155859  -0.25485513
0.05424322 -0.27320358]
Mean accuracy: 0.6753246753246753

Using KerasClassifier :

from numpy import loadtxt
import numpy as np
from scikeras.wrappers import KerasClassifier
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import accuracy_score
from sklearn.utils import shuffle
from sklearn.pipeline import make_pipeline

def model_nn(num_sample=8, num_hidden=12):

    model = Sequential()
    model.add(Dense(num_hidden, input_shape=(num_sample,), activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['binary_crossentropy'])
    return model

# load the dataset
dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
modelscikeras = KerasClassifier(model_nn(8, 12), epochs=30, batch_size=16, verbose=False)

n_split = 10
X, y = dataset[:, 0:8],  dataset[:, 8].astype(int)
num_runs = 5
all_scores = []
kf = KFold(n_splits=n_split, shuffle=True)
for train_index, eval_index in kf.split(X, y):
    x_test, x_train = X[eval_index, :], X[train_index, :] 
    y_test, y_train = y[eval_index], y[train_index]
    scores = []
    score = 0
    for nrun in range(num_runs):
        biais = modelscikeras.model.layers[1].get_weights()[1]
        print(f"Before: {biais}")
        modelscikeras.fit(x_, y_)
        biais = modelscikeras.model.layers[1].get_weights()[1]
        print(f"After: {biais}")
        y_pred = modelscikeras.predict(x_test)
        score += np.mean(accuracy_score(y_test, y_pred))
    score /= num_runs
    all_scores.append(score)
print(f"{Mean accuracy: np.mean(all_scores)}")

Console:

Before: [0. 0. 0. 0. 0. 0. 0. 0.]
After: [-0.11853977  0.         -0.13868926 -0.05590332  0.18646333 -0.02861278
0.02369083 -0.05153953]
Before: [-0.11853977  0.         -0.13868926 -0.05590332  0.18646333 -0.02861278
0.02369083 -0.05153953]
After: [-0.13340724  0.         -0.22247441 -0.05590332  0.4105426  -0.03854674
0.01876416 -0.05754992]
Before: [-0.13340724  0.         -0.22247441 -0.05590332  0.4105426  -0.03854674
0.01876416 -0.05754992]
After: [-0.15757883  0.         -0.30742666 -0.06423395  0.6475376  -0.04791947
0.0227307  -0.02398843]
Before: [-0.15757883  0.         -0.30742666 -0.06423395  0.6475376  -0.04791947
0.0227307  -0.02398843]
After: [-0.27340996  0.         -0.4003844  -0.0724286   0.8939626  -0.05549236
0.01277073 -0.01975848]
Before: [-0.27340996  0.         -0.4003844  -0.0724286   0.8939626  -0.05549236
0.01277073 -0.01975848]
After: [-0.41049564  0.         -0.4966296  -0.09285609  1.1078017  -0.04804759
0.01173483 -0.03017891]
Mean accuracy: 0.6519480519480518
Before: [-0.41049564  0.         -0.4966296  -0.09285609  1.1078017  -0.04804759
0.01173483 -0.03017891]
After: [-0.5412145   0.         -0.5893507  -0.11136947  1.3047545  -0.06426006
0.02845561 -0.03688649]
Before: [-0.5412145   0.         -0.5893507  -0.11136947  1.3047545  -0.06426006
0.02845561 -0.03688649]
After: [-0.66179794  0.         -0.6727742  -0.11238873  1.4997679  -0.0352393
0.0333738  -0.03578174]
Before: [-0.66179794  0.         -0.6727742  -0.11238873  1.4997679  -0.0352393
0.0333738  -0.03578174]
After: [-0.8112038   0.         -0.7471358  -0.12658192  1.6813278  -0.03301082
0.05713192 -0.05601563]
Before: [-0.8112038   0.         -0.7471358  -0.12658192  1.6813278  -0.03301082
0.05713192 -0.05601563]
After: [-0.97863823  0.         -0.79813385 -0.1411471   1.8627418  -0.04511299
0.07826705 -0.0842851 ]
Before: [-0.97863823  0.         -0.79813385 -0.1411471   1.8627418  -0.04511299
0.07826705 -0.0842851 ]
After: [-1.1294042   0.         -0.87194943 -0.16817145  2.0405526  -0.03309529
0.10685121 -0.10747109]
Mean accuracy: 0.6571428571428571

I am using:

Windows 10
Scikeras 0.10.0
TensorFlow 2.11.0
keras 2.11.0

Why do you think about these weights and the learned tendency that takes place in the 2nd case? Do you know a way to see if the model is properly reset when fitting with KerasClassifier ? Thank you in advance for your time

[adriangb]

You're using a pre-trained model:

KerasClassifier(model_nn(8, 12))

I think what you want is:

KerasClassifier(model_nn, num_sample=8, num_hidden=12)

[nabilalibou]

So it was this all along, thank you very much