HP Tuning with keras and k-fold cross validation

[datistiquo]

I have no detailed knowledge of the GCP inner working. When exactly is HP Tuning"called" when using keras. If you use a validation accuracy is it first performed when model.evaluate() is called? I asked my self because I want to do cross-validation with a keras model (not tf.keras if this matters).

Like:

from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import StratifiedKFold
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# load pima indians dataset
dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
# define 10-fold cross validation test harness
kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
cvscores = []
for train, test in kfold.split(X, Y):
  # create model
    model = Sequential()
    model.add(Dense(12, input_dim=8, activation='relu'))
    model.add(Dense(8, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    # Compile model
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    # Fit the model
    model.fit(X[train], Y[train], epochs=150, batch_size=10, verbose=0)
    # evaluate the model
    scores = model.evaluate(X[test], Y[test], verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1]*100))
    cvscores.append(scores[1] * 100)
print("%.2f%% (+/- %.2f%%)" % (numpy.mean(cvscores), numpy.std(cvscores)))
val_acc=numpy.mean(cvscores)

from https://machinelearningmastery.com/evaluate-performance-deep-learning-models-keras/

Since in the for loop each time fit and evaluate are called I wonder if this has an impact for HP tuning with GCP Also, my metric wanted for the HP Tuning would not be the single validation acc in each step but the mean after the loop as I appended at the end. Can I do that to assign a custom metric like the average one above to the desired HP Tuning-Metric? Maybe this does matter at all since HP Tuning focuses only on the metric name in the HPconfig file. But why I ask is performance issue if in each step of the loop first a WHOLE HP Tuning for each single model inside the loop is done?

[sirtorry]

Hi @datistiquo, thank you for your question. Could you please identify which sample you used?

[datistiquo]

It is not specified to a sample but in general?

[andrewferlitsch]

datistiquo,

Automated HP Tuning is part of AI-Platform (formerly CMLE). If you are running as a standalone job on GCP, then you have to do you own HP tuning. General information can be found at this link: https://cloud.google.com/ml-engine/docs/tensorflow/hyperparameter-tuning-overview

You can define custom metrics for evaluation in Keras. It's near the end on this web page: https://keras.io/metrics/