Keras Tuner for time series split

[vignesh0710]

Is it possible to use Keras tuner for tuning a NN using Time Series Split , similar to sklearn.model_selection.TimeSeriesSplit in sklearn.

For example consider a sample tuner class from https://towardsdatascience.com/hyperparameter-tuning-with-keras-tuner-283474fbfbe

from kerastuner import HyperModel
class SampleModel(HyperModel):
    def __init__(self, input_shape):
        self.input_shape = input_shape
    def build(self, hp):
        model = Sequential()
        model.add(
            layers.Dense(
                units=hp.Int('units', 8, 64, 4, default=8),
                activation=hp.Choice(
                    'dense_activation',
                    values=['relu', 'tanh', 'sigmoid'],
                    default='relu'),
                input_shape=input_shape
            )
        )

        model.add(layers.Dense(1))

        model.compile(
            optimizer='rmsprop',loss='mse',metrics=['mse']
        )

        return model

tuner:

tuner_rs = RandomSearch(
            hypermodel,
            objective='mse',
            seed=42,
            max_trials=10,
            executions_per_trial=2)

tuner_rs.search(x_train_scaled, y_train, epochs=10, validation_split=0.2, verbose=0)

So instead of validation_split = 0.2, in the above line is it possible to do the following

from sklearn.model_selection import TimeSeriesSplit

#defining a time series split object
tscv = TimeSeriesSplit(n_splits = 5)

#using that in Keras Tuner
tuner_rs.search(x_train, y_train, epochs=10, validation_split=tscv, verbose=0)

[haifeng-jin]

I don't think it is currently possible within keras-tuner. However, you can always provide your own data and validation data instead of using the split.

[PiotrAntoniak]

Not sure if its allowed to link the kaggle notebooks (you can find implementations by just googling jane street competition) from here so the code goes:

class CVTuner(kt.engine.tuner.Tuner):
    def run_trial(self, trial, X, y, splits, batch_size=32, epochs=1,callbacks=None):

        val_losses = []
        for train_indices, test_indices in splits:
            X_train, X_test = [x[train_indices] for x in X], [x[test_indices] for x in X]
            y_train, y_test = [a[train_indices] for a in y], [a[test_indices] for a in y]
            if len(X_train) < 2:
                X_train = X_train[0]
                X_test = X_test[0]
            if len(y_train) < 2:
                y_train = y_train[0]
                y_test = y_test[0]

            model = self.hypermodel.build(trial.hyperparameters)
            hist = model.fit(X_train,y_train,
                      validation_data=(X_test,y_test),
                      epochs=epochs,
                        batch_size=batch_size,
                      callbacks=callbacks)

            val_losses.append([hist.history[k][-1] for k in hist.history])
        val_losses = np.asarray(val_losses)
        self.oracle.update_trial(trial.trial_id, {k:np.mean(val_losses[:,i]) for i,k in enumerate(hist.history.keys())})
        self.save_model(trial.trial_id, model)

the split is provided by this class: https://gist.github.com/terminate9298/917f5c0e70e58703215ab858f1adb7d3

[Tiger-QU]

I don't think it is currently possible within keras-tuner. However, you can always provide your own data and validation data instead of using the split.

Is there an update on this as of now?