Chapter 10 runtime error from grid search neural network paramaters

[nickcorona]

keras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)
keras_reg.fit(
    X_train,
    y_train,
    epochs=100,
    validation_data=(X_valid, y_valid),
    callbacks=[keras.callbacks.EarlyStopping(patience=10)],
)
mse_test = keras_reg.score(X_test, y_test)

param_distribs = {
    "n_hidden": [0, 1, 2, 3],
    "n_neurons": np.arange(1, 100),
    "learning_rate": reciprocal(3e-4, 3e-2),
}

rnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=10, cv=3)
rnd_search_cv.fit(
    X_train,
    y_train,
    epochs=100,
    validation_data=(X_valid, y_valid),
    callbacks=[keras.callbacks.EarlyStopping(patience=10)],
)

This code yields this error:

Traceback (most recent call last):
  File "<stdin>", line 6, in <module>
  File "/home/nick/.cache/pypoetry/virtualenvs/hml-EoLX4ttF-py3.7/lib/python3.7/site-packages/sklearn/model_selection/_search.py", line 736, in fit
    **self.best_params_))
  File "/home/nick/.cache/pypoetry/virtualenvs/hml-EoLX4ttF-py3.7/lib/python3.7/site-packages/sklearn/base.py", line 82, in clone
    (estimator, name))
RuntimeError: Cannot clone object <tensorflow.python.keras.wrappers.scikit_learn.KerasRegressor object at 0x7f1c2c5e54d0>, as the constructor either does not set or modifies parameter learning_rate

Any idea why?

[lord8266]

I can't reproduce this on

scikit-learn             0.21.3   
scipy                    1.3.1    
tensorflow-gpu           2.0.0  

But I do get the same error here: keras-team/keras/#13586,

[furian-rising]

I've had similar issues and commented the following on #keras-team/keras, pasted below for convenience.

Under sklearn.model_selection._search.py (sklearn version 0.22.1) line 735, nested clone call seems to be responsible:

self.best_estimator_ = clone(clone(base_estimator).set_params(
**self.best_params_))

seems to be the culprit. It changed from the following in a previous version:

self.best_estimator_ = clone(base_estimator).set_params(
**self.best_params_)

Changing this back to the previous version seems to fix the error (Cannot clone object...), though not sure that is the correct solution... based upon the discussion in this thread.

Thoughts??

[furian-rising]

Here's what I ended up doing: https://github.com/scikit-learn/scikit-learn/issues/15722#issuecomment-589264317

[jkmackie]

The quick fix for the "cannot clone object" error is the following: downgrade scikit-learn from 0.22.x to 0.21.3. Here is my Anaconda setup:

scikit-learn          0.21.3
scipy                  1.4.1    
tensorflow-gpu         2.1.0 

[ageron]

Thanks for your feedback, and thanks to everyone for investigating this issue. I added the following comment to the notebook:

Warning: the following cell crashes at the end of training. This seems to be caused by Keras issue #13586, which was triggered by a recent change in Scikit-Learn. Pull Request #13598 seems to fix the issue, so this problem should be resolved soon.

Hoping that it does indeed get resolved soon.

[LiuXiaosxy]

Hi, I had a similar but different problem, I had the following problem while using hyperparameter search via sklearn:

def build_model(optimizer):
    grid_model = Sequential()
    grid_model.add(LSTM(50, return_sequences=True,input_shape=(30,5)))
    grid_model.add(LSTM(50))
    grid_model.add(Dropout(0.2))
    grid_model.add(Dense(1))

    grid_model.compile(optimizer=optimizer ,loss='mse')
    return grid_model

grid_model = KerasRegressor(build_fn = build_model,verbose = 1,validation_data = (testX,testY))
parameters = {'batch_size' : [16,20],'epochs': [8,10],'optimizer': ['adam','Adadelta']}

grid_search = GridSearchCV(estimator = grid_model, param_grid = parameters,cv = 2)

grid_search = grid_search.fit(trainX, trainY)

The error is ：RuntimeError: Cannot clone object <tensorflow.python.keras.wrappers.scikit_learn.KerasRegressor object at 0x000001BFDD4AF4C0>, as the constructor either does not set or modifies parameter validation_data