Model performance difference when using KerasClassifier

[nabilalibou]

Hello,

I have a rather strange bug: Wrapping a model with KerasClassifier will decrease its performance (binary accuracy) by about 10%.

from numpy import loadtxt
import numpy as np
from scikeras.wrappers import KerasClassifier
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import accuracy_score
from sklearn.utils import shuffle
from sklearn.pipeline import make_pipeline

def model_nn():

    model = Sequential()
    model.add(Dense(12, input_shape=(8,), activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['binary_crossentropy'])
    return model

# load the dataset
dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
modelscikeras = KerasClassifier(model_nn, epochs=30, batch_size=16, verbose=False)

n_split = 10
X, y = dataset[:, 0:8],  dataset[:, 8].astype(int)
num_runs = 30
all_scores = []
kf = KFold(n_splits=n_split, shuffle=True)
for train_index, eval_index in kf.split(X, y):
    x_test, x_train = X[eval_index, :], X[train_index, :]  # shuffle
    y_test, y_train = y[eval_index], y[train_index]
    scores = []
    score = 0
    for nrun in range(num_runs):
        modelscikeras.fit(x_, y_)
        y_pred = modelscikeras.predict(x_test)
        score += np.mean(accuracy_score(y_test, y_pred))
    score /= num_runs
    all_scores.append(score)
print(np.mean(all_scores))

The code above will consistently gives me an average accuracy across 30 runs * 10 kFold split of 67% binary accuracy.

Whereas this code without using KerasClassifier:

from numpy import loadtxt
import numpy as np
from scikeras.wrappers import KerasClassifier
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import accuracy_score
from sklearn.utils import shuffle
from sklearn.pipeline import make_pipeline

def model_nn():

    model = Sequential()
    model.add(Dense(12, input_shape=(8,), activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['binary_crossentropy'])
    return model

# load the dataset
dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
model_nn = model_nn()

n_split = 10
X, y = dataset[:, 0:8],  dataset[:, 8].astype(int)
num_runs = 30
all_scores = []
kf = KFold(n_splits=n_split, shuffle=True)
for train_index, eval_index in kf.split(X, y):
    x_test, x_train = X[eval_index, :], X[train_index, :]  # shuffle
    y_test, y_train = y[eval_index], y[train_index]
    scores = []
    score = 0
    for nrun in range(num_runs):
        x_, y_ = shuffle(x_train, y_train)
        model_nn.fit(x_, y_, epochs=30, batch_size=16, verbose=False)
        y_pred = (model_nn.predict(x_test) > 0.5).astype(int)
        score += np.mean(accuracy_score(y_test, y_pred))
    score /= num_runs
    all_scores.append(score)
print(np.mean(all_scores))

Will consistently gives me 78% of binary accuracy.

I tried to put the model out of the function and define it directly in the script, to put the compilation outside of model_nn(), to put the compilation parameters in KerasClassifier() etc... In any combination wrapping the model in KerasClassifier, fitting and predicting with it gives me less accuracy. What do I not see ? Thank you in advance for your time

I am using:

• Windows 10
• Scikeras 0.10.0
• TensorFlow 2.11.0
• keras 2.11.0

[nabilalibou]

It is because fitting a KerasClassifier reset the neural network weights unlike fitting a Keras model. I am closing the issue.

[adriangb]

I was on vacation but was just about to look into this! Sorry for the delay.

fitting a KerasClassifier reset the neural network weights unlike fitting a Keras model

Indeed in the scikit-learn API calling fit() resets the model. Maybe you want partial_fit or warm_start?