Add tutorial and doc for Custom Objective function

[AnconaAndrea]

I am implementing a classifier with three classes, I am using hot encoding for the labels

I want to use a custom objective function in the tuner (precision at class 1):

I defined:

def prec_class1(y_true, y_pred): from sklearn.metrics import precision_recall_curve threshold=0.76 y_pred = np.squeeze(y_pred, axis=1) y_true = np.squeeze(ytrue, axis=1) precision, recall, = precision_recall_curve(y_true[:,1],y_pred[:,1]) for m in range(len(recall)): if recall[m] > threshold and recall[m] < threshold + 0.001: prec = precision[m] return Threshold

and then:

tuner1 = Hyperband(hypermodel, objective= kt.Objective("prec_class1", direction="max"), max_epochs=30, executions_per_trial=1, #nr di volte in cui la stessa configurazione viene testata directory=root_dir)

do you think this would work?

Thank you

[ben-arnao]

See below. "score" can be whatever you want, it doesn't even have to be called "score". It is up to you whatever value you want to optimize. You will just report the value in run_trial, and the make sure the name matches what you put in your objective declaration.. Objective just defines what to optimize and which direction to optimize in.

def run_trial(self, trial, ...): score = ... self.oracle.update_trial(trial.trial_id, {'score': score})

On Tue, Mar 24, 2020 at 2:59 PM andreaAnc notifications@github.com wrote:

I am implementing a classifier with three classes, I am using hot encoding for the labels

I want to use a custom objective function in the tuner (precision at class 1):

I defined:

def prec_class1(y_true, y_pred): from sklearn.metrics import precision_recall_curve threshold=0.76 y_pred = np.squeeze(y_pred, axis=1) y_true = np.squeeze(ytrue, axis=1) precision, recall, = precision_recall_curve(y_true[:,1],y_pred[:,1]) for m in range(len(recall)): if recall[m] > threshold and recall[m] < threshold + 0.001: prec = precision[m] return Threshold

and then:

tuner1 = Hyperband(hypermodel, objective= kt.Objective("prec_class1", direction="max"), max_epochs=30, executions_per_trial=1, #nr di volte in cui la stessa configurazione viene testata directory=root_dir)

do you think this would work?

Thank you

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/keras-team/keras-tuner/issues/263, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB5OIMIPZWQAOBS53OB2RQDRJD7JFANCNFSM4LS5FFBQ .

[AnconaAndrea]

Thank you very much for your reply.

Where should I define def run_trial(self, trial, ...): score = ... self.oracle.update_trial(trial.trial_id, {'score': score}) ?

Within the main code?

and in this way?

def run_trial(self, trial,y_true,y_pred): from sklearn.metrics import precision_recall_curve threshold=0.76 y_pred = np.squeeze(y_pred, axis=1) y_true = np.squeeze(ytrue, axis=1) precision, recall, = precision_recall_curve(y_true[:,1],y_pred[:,1]) for m in range(len(recall)): if recall[m] > threshold and recall[m] < threshold + 0.001: score = precision[m] self.oracle.update_trial(trial.trial_id, {'score': score}

[ben-arnao]

@andreaAnc

What you will do is override the run_trial method and provide your own implementation of score.

It is up to you how you want to assign a score to the model.

model -> score

class MyTuner(kt.Tuner):

    def run_trial(self, trial, ...):
        model = self.hypermodel.build(trial.hyperparameters)
        score = ...
        self.oracle.update_trial(trial.trial_id, {'score': score})
        self.oracle.save_model(trail.trial_id, model)

For example you could define your custom metric, provide it to the keras fit() function then retrieve the metric for the last epoch with the history api. If that's what you want to do.

Take that value and put it in the trials' score.

[AnconaAndrea]

do you mean override the run_trial method here? https://github.com/keras-team/keras-tuner/blob/e06bec6cac2172bd2cb038cca645578ef2ad307a/kerastuner/engine/base_tuner.py#L109

[ben-arnao]

@andreaAnc As the code snippet shows above just subclass a tuner and define your own run_trial method. In here you are basically supply a set of parameters and you need to assign a score to them. How you want to do that is up to you.

[Shivayv]

Can you share some sort of a snippet ? I am trying to use AUC as a objective function for binary classification.

Details in link : https://stackoverflow.com/questions/61080146/kerastuner-custom-objective-function

[ben-arnao]

@Shivayv

Since any params in search() are passed to fit(), this is basically your fit function. All you have to do here is define you metric, include it in fit. Loss and any metrics are all available to you as possible objectives (without having to manually score your trial) and you can specific by name.

def my_custom_metric(pred, true)
    return ....

tuner = RandomSearch(
    hypermodel,
    objective='my_custom_metric',
    max_trials=10,
    directory='my_dir',
    project_name='helloworld')

tuner.search(x, y,
             epochs=5,
             validation_data=(val_x, val_y))
             metrics=[my_custom_metric]

[Shivayv]

Thanks but I still get the error on executing the following code :

from sklearn import metrics
from keras import backend as K

def auc(y_true, y_pred):
    auc = tf.metrics.auc(y_true, y_pred)[1]
    K.get_session().run(tf.local_variables_initializer())
    return auc

EXECUTIONS_PER_TRIAL = 5

b_tuner = BayesianOptimization(
    tune_nn_model,
    objective='auc',
    max_trials=MAX_TRIALS,
    executions_per_trial=EXECUTIONS_PER_TRIAL,
    directory='test_dir101897',
    project_name='b_tune_nn',
    seed=12347
)

ValueError: Could not infer optimization direction ("min" or "max") for unknown metric "auc". Please specify the objective  asa `kerastuner.Objective`, for example `kerastuner.Objective("auc", direction="min")`. 

[ben-arnao]

@Shivayv That's because you need to supply a direction since it can't be inferred.

Also my mistake metrics needs to be supplied in compile() not fit()

from tensorflow import keras
from tensorflow.keras import layers
from kerastuner.tuners import RandomSearch
import tensorflow as tf
import kerastuner

def build_model(hp):
    model = keras.Sequential()
    model.add(layers.Dense(units=hp.Int('units',
                                        min_value=32,
                                        max_value=512,
                                        step=32),
                           activation='relu'))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate',
                      values=[1e-2, 1e-3, 1e-4])),
        loss='binary_crossentropy',
        metrics=[tf.keras.metrics.AUC()])
    return model

tuner = RandomSearch(
    build_model,
    objective=kerastuner.Objective("auc", direction="min"),
    max_trials=5,
    executions_per_trial=3,
    directory='my_dir',
    project_name='helloworld')

x = [[1], [2], [3]]
y = [[0], [1], [0]]

tuner.search(x, y, epochs=5)

[tolandwehr]

@ben-arnao

Hi there!

Trying to implement this structure, but I get an argument error

def CustomVariousHT(y_true, y_pred):
    if np.isnan(y_pred).any():
        MAD= 1000000
    else:
        MAD=median_absolute_error(y_true, y_pred)
        print(MAD)
    return MAD

class LSTMHyperModel(HyperModel):
    def __init__(self, input_shape):
        self.input_shape = input_shape

    def build(self, hp):
        model = keras.Sequential()
        .
        .
        .
        model.compile(
            optimizer=keras.optimizers.Adam(
                hp.Float(
                    'learning_rate',
                    min_value=1e-4,
                    max_value=1e-2,
                    sampling='LOG',
                    default=1e-3
                )
            ),
            loss=hp.Choice(
                'Loss',
                values=['logcosh','mae', 'mse'],
            ),
            metrics=[CustomVariousHT()]
        )
        return model

from kerastuner.tuners import BayesianOptimization, Hyperband
import kerastuner

hypermodel = LSTMHyperModel(input_shape=(X_train.shape[1],X_train.shape[2]))

class MyTuner(Hyperband):
  def run_trial(self, trial, *args, **kwargs):
    kwargs['batch_size'] = trial.hyperparameters.Int('batch_size', 32, 256, step=32)
    kwargs['epochs'] = trial.hyperparameters.Int('epochs', 7, 37, step=5)
    super(MyTuner, self).run_trial(trial, *args, **kwargs)

tuner = MyTuner(
    hypermodel,
    objective=kerastuner.Objective("MAD", direction="min"),
    max_epochs=37,
    seed=737,
    executions_per_trial=3,
    directory='.',
    project_name='KerasTuner4')

At initializing MyTuner this throws the error

Traceback (most recent call last):
  File "C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\kerastuner\engine\hypermodel.py", line 105, in build
    model = self.hypermodel.build(hp)
  File "<ipython-input-23-57edfab04d29>", line 122, in build
    metrics=[CustomVariousHT()]
TypeError: CustomVariousHT() missing 2 required positional arguments: 'y_true' and 'y_pred'

Apparently, it fails to hand over the prediction and the true training value. Any instruction on how to declare this? Any hint deeply appreciated.

[ambaresthete]

@tolandwehr Your metric should not be called, it has to be passed as a parameter. Check below - model.compile( optimizer=keras.optimizers.Adam( hp.Float( 'learning_rate', min_value=1e-4, max_value=1e-2, sampling='LOG', default=1e-3 ) ), loss=hp.Choice( 'Loss', values=['logcosh','mae', 'mse'], ), metrics=[CustomVariousHT] )

Remove the '()' after CustomVariousHT.

[tolandwehr]

@ambaresthete @ben-arnao

Okay, without the brackets it's trying to accept the custom code, but throws errors regarding tf and numpy harmonization.

Here the brief and full error line. Pops with and without the y_pred check for NaNs in the prediction tensor.

OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

---------------------------------------------------------------------------
OperatorNotAllowedInGraphError            Traceback (most recent call last)
<ipython-input-132-6b795f793875> in <module>
----> 1 tuner.search(X_train, y_train, epochs=38, validation_split=0.1, callbacks=[tf.keras.callbacks.EarlyStopping('val_loss', patience=3)])

~\Anaconda3\envs\Tensorflow\lib\site-packages\kerastuner\engine\base_tuner.py in search(self, *fit_args, **fit_kwargs)
    128 
    129             self.on_trial_begin(trial)
--> 130             self.run_trial(trial, *fit_args, **fit_kwargs)
    131             self.on_trial_end(trial)
    132         self.on_search_end()

<ipython-input-130-0c5168f83bfd> in run_trial(self, trial, *args, **kwargs)
      5     kwargs['batch_size'] = trial.hyperparameters.Int('batch_size', 32, 256, step=32)
      6     kwargs['epochs'] = trial.hyperparameters.Int('epochs', 3, 38, step=5)
----> 7     super(MyTuner, self).run_trial(trial, *args, **kwargs)

~\Anaconda3\envs\Tensorflow\lib\site-packages\kerastuner\tuners\hyperband.py in run_trial(self, trial, *fit_args, **fit_kwargs)
    385             fit_kwargs['epochs'] = hp.values['tuner/epochs']
    386             fit_kwargs['initial_epoch'] = hp.values['tuner/initial_epoch']
--> 387         super(Hyperband, self).run_trial(trial, *fit_args, **fit_kwargs)
    388 
    389     def _build_model(self, hp):

~\Anaconda3\envs\Tensorflow\lib\site-packages\kerastuner\engine\multi_execution_tuner.py in run_trial(self, trial, *fit_args, **fit_kwargs)
     94 
     95             model = self.hypermodel.build(trial.hyperparameters)
---> 96             history = model.fit(*fit_args, **copied_fit_kwargs)
     97             for metric, epoch_values in history.history.items():
     98                 if self.oracle.objective.direction == 'min':

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\keras\engine\training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
   1096                 batch_size=batch_size):
   1097               callbacks.on_train_batch_begin(step)
-> 1098               tmp_logs = train_function(iterator)
   1099               if data_handler.should_sync:
   1100                 context.async_wait()

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py in __call__(self, *args, **kwds)
    778       else:
    779         compiler = "nonXla"
--> 780         result = self._call(*args, **kwds)
    781 
    782       new_tracing_count = self._get_tracing_count()

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
    821       # This is the first call of __call__, so we have to initialize.
    822       initializers = []
--> 823       self._initialize(args, kwds, add_initializers_to=initializers)
    824     finally:
    825       # At this point we know that the initialization is complete (or less

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py in _initialize(self, args, kwds, add_initializers_to)
    695     self._concrete_stateful_fn = (
    696         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 697             *args, **kwds))
    698 
    699     def invalid_creator_scope(*unused_args, **unused_kwds):

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2853       args, kwargs = None, None
   2854     with self._lock:
-> 2855       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2856     return graph_function
   2857 

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   3211 
   3212       self._function_cache.missed.add(call_context_key)
-> 3213       graph_function = self._create_graph_function(args, kwargs)
   3214       self._function_cache.primary[cache_key] = graph_function
   3215       return graph_function, args, kwargs

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   3073             arg_names=arg_names,
   3074             override_flat_arg_shapes=override_flat_arg_shapes,
-> 3075             capture_by_value=self._capture_by_value),
   3076         self._function_attributes,
   3077         function_spec=self.function_spec,

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    984         _, original_func = tf_decorator.unwrap(python_func)
    985 
--> 986       func_outputs = python_func(*func_args, **func_kwargs)
    987 
    988       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\eager\def_function.py in wrapped_fn(*args, **kwds)
    598         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    599         # the function a weak reference to itself to avoid a reference cycle.
--> 600         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    601     weak_wrapped_fn = weakref.ref(wrapped_fn)
    602 

~\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\framework\func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

OperatorNotAllowedInGraphError: in user code:

    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\keras\engine\training.py:806 train_function  *
        return step_function(self, iterator)
    <ipython-input-128-d8f81f289cfd>:6 CustomVariousHT  *
        MAD=median_absolute_error(y_true, y_pred)
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\sklearn\utils\validation.py:355 inner_f  *
        return f(**kwargs)
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\sklearn\metrics\_regression.py:392 median_absolute_error  *
        y_type, y_true, y_pred, multioutput = _check_reg_targets(
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\sklearn\metrics\_regression.py:84 _check_reg_targets  *
        check_consistent_length(y_true, y_pred)
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\sklearn\utils\validation.py:253 check_consistent_length  *
        uniques = np.unique(lengths)
    <__array_function__ internals>:6 unique  **

    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\numpy\lib\arraysetops.py:263 unique
        ret = _unique1d(ar, return_index, return_inverse, return_counts)
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\numpy\lib\arraysetops.py:311 _unique1d
        ar.sort()
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\framework\ops.py:877 __bool__
        self._disallow_bool_casting()
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\framework\ops.py:487 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    C:\Users\alias\Anaconda3\envs\Tensorflow\lib\site-packages\tensorflow\python\framework\ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

[formigone]

I was able to get this to work following the answer by ben-arnao.

import tensorflow as tf

from kerastuner import Objective, HyperParameters
from kerastuner.tuners import RandomSearch

def build_model(hp: HyperParameters) -> tf.keras.Model:
    input = tf.keras.Input(shape=(1,))
    x = input
    for i in range(hp.Int('Layers', min_value=2, max_value=4, step=1)):
        dim = hp.Int(f'Layers_{i + 1}Dim', min_value=2, max_value=4, step=1)
        x = tf.keras.layers.Dense(dim, activation=tf.nn.relu, name=f'Dense_{i + 1}x{dim}')(x)

    outputs = tf.keras.layers.Dense(1, activation=tf.nn.sigmoid, name='OutputSigmoid')(x)

    model = tf.keras.Model(inputs=input, outputs=outputs)
    model.compile(optimizer=tf.keras.optimizers.Adam(lr=1e-5),
                  loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
                  metrics=[
                      tf.keras.metrics.TruePositives(name='tp'),
                      tf.keras.metrics.FalsePositives(name='fp'),
                      tf.keras.metrics.TrueNegatives(name='tn'),
                      tf.keras.metrics.FalseNegatives(name='fn'),
                      tf.keras.metrics.BinaryAccuracy(name='accuracy'),
                      tf.keras.metrics.Precision(name='precision'),
                      tf.keras.metrics.Recall(name='recall'),
                      tf.keras.metrics.AUC(name='auc'),
                  ])
    return model

tuner = RandomSearch(
    build_model,
    objective=Objective('val_auc', direction='max'),
    max_trials=10,
    executions_per_trial=5,
    directory='../.data/kt',
    project_name='helloworld')

tuner.search(train_data.batch(TRAIN_BATCH_SIZE),
             epochs=5,
             shuffle=True,
             class_weight=class_weight,
             validation_data=dev_data.batch(1024))

[akbaramed]

If any one is still looking around. To create a custom error metric function to use during Hyper parameter tuning

example want to use F1-score to find best Hyr Param

1. Create the function

import keras.backend as K

def f1_score(y_true, y_pred): true_positives = K.sum(K.round(K.clip(y_true y_pred, 0, 1))) possible_positives = K.sum(K.round(K.clip(y_true, 0, 1))) predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1))) precision = true_positives / (predicted_positives + K.epsilon()) recall = true_positives / (possible_positives + K.epsilon()) f1_scr = 2(precision*recall)/(precision+recall+K.epsilon()) return f1_scr

2. Add function to model compile

Modell.compile(metrics = [....., metrics.FalseNegatives(), f1_score])

3. declare the custom metric in tuner

from kerastuner import HyperParameters, Objective

tuner = Hyperband( model_create, max_epochs=15, objective=Objective('f1_score', direction='max'), ........) ########

[albertosnchz]

compile(metrics = [....., metrics.FalseNegatives(), f1_score])

I still have the same problem and I can't solve it!

https://stackoverflow.com/q/65275565/14346588

[albertosnchz]

I am using Keras Tuner to optimize a CNN model for a regression problem. Basicly I have sequences of DNA that I turned into a matrix in order to use them as images to train a CNN model. What I want to predict is a percentage that depends on those sequences.

this is my model to try:

def build_model(hp):  
  model = keras.Sequential([
    keras.layers.Conv2D(
        filters=hp.Int('conv_1_filter', min_value=32, max_value=128, step=16),
        kernel_size=hp.Choice('conv_1_kernel', values = [3,3]),
        activation='relu',
        padding = 'same',
        input_shape=(30,4,1)
    ),
    keras.layers.Conv2D(
        filters=hp.Int('conv_2_filter', min_value=32, max_value=96, step=16),
        kernel_size=hp.Choice('conv_2_kernel', values = [5,3]),
        activation='relu',
        padding = 'same'
    ),
    keras.layers.Conv2D(
        filters=hp.Int('conv_3_filter', min_value=32, max_value=64, step=16),
        kernel_size=hp.Choice('conv_3_kernel', values = [7,3]),
        activation='relu',
        padding = 'same'
    ),
    keras.layers.MaxPooling2D(
        pool_size=hp.Choice('maxpool_1_size', values = [2,2]),
    ),
    keras.layers.Flatten(),
    keras.layers.Dense(
        units=hp.Int('dense_1_units', min_value=32, max_value=128, step=16),
        activation='relu'
    ),
    keras.layers.Dense(1, activation='relu')
  ])

  model.compile(optimizer=keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3,1e-4])),
              loss='mean_absolute_error',
              metrics=[R2_Score])

  return model

As I am working with regression I used this fuction for R2Score as in Keras they lack of it:

def R2_Score(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

For tunning I am using this code:

tuner=RandomSearch(build_model,
                   objective= kerastuner.Objective('R2_Score', direction='max'),
                   max_trials=5,
                   directory='output',
                   project_name="CRISPR-Cas9")

tuner.search(X_train, y_train,
             epochs=5,
             validation_data=(X_test, y_test),
             metrics=[R2_Score])

I get the error in model.compile, and I suspect that I will get the same in the tuner or tuner.search:

INFO:tensorflow:Reloading Oracle from existing project output/CRISPR-Cas9/oracle.json

Search: Running Trial #1

Hyperparameter    |Value             |Best Value So Far 
conv_1_filter     |64                |?                 
conv_1_kernel     |3                 |?                 
conv_2_filter     |48                |?                 
conv_2_kernel     |3                 |?                 
conv_3_filter     |32                |?                 
conv_3_kernel     |7                 |?                 
dense_1_units     |96                |?                 
learning_rate     |0.0001            |?                 

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-72-0de1369470a1> in <module>()
      8              epochs=5,
      9              validation_data=(sequences_test, indel_test),
---> 10              metrics=['R2_Score'])

3 frames
/usr/local/lib/python3.6/dist-packages/kerastuner/engine/base_tuner.py in search(self, *fit_args, **fit_kwargs)
    129 
    130             self.on_trial_begin(trial)
--> 131             self.run_trial(trial, *fit_args, **fit_kwargs)
    132             self.on_trial_end(trial)
    133         self.on_search_end()

/usr/local/lib/python3.6/dist-packages/kerastuner/engine/multi_execution_tuner.py in run_trial(self, trial, *fit_args, **fit_kwargs)
     94             copied_fit_kwargs['callbacks'] = callbacks
     95 
---> 96             history = self._build_and_fit_model(trial, fit_args, copied_fit_kwargs)
     97             for metric, epoch_values in history.history.items():
     98                 if self.oracle.objective.direction == 'min':

/usr/local/lib/python3.6/dist-packages/kerastuner/engine/tuner.py in _build_and_fit_model(self, trial, fit_args, fit_kwargs)
    139         """
    140         model = self.hypermodel.build(trial.hyperparameters)
--> 141         return model.fit(*fit_args, **fit_kwargs)
    142 
    143     def run_trial(self, trial, *fit_args, **fit_kwargs):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
    106   def _method_wrapper(self, *args, **kwargs):
    107     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
--> 108       return method(self, *args, **kwargs)
    109 
    110     # Running inside `run_distribute_coordinator` already.

TypeError: fit() got an unexpected keyword argument 'metrics'

I saw some solutions for this problem but none work form me, also almost every example of CNN I see is used in clasification but I am doing a regression.

Can anybody give me a hand?

[akbaramed]

Hey, There is no metrics keyword in model.fit or tuner.search remove it from tuner.search.

Fyi I use Google Colab. If the error still doesnt go then check the versions of keras, tensorflow as well.

On Sun, Dec 13, 2020, 9:40 PM albertosnchz notifications@github.com wrote:

I am using Keras Tuner to optimize a CNN model for a regression problem. Basicly I have sequences of DNA that I turned into a matrix in order to use them as images to train a CNN model. What I want to predict is a percentage that depends on those sequences.

this is my model to try:

def build_model(hp): model = keras.Sequential([ keras.layers.Conv2D( filters=hp.Int('conv_1_filter', min_value=32, max_value=128, step=16), kernel_size=hp.Choice('conv_1_kernel', values = [3,3]), activation='relu', padding = 'same', input_shape=(30,4,1) ), keras.layers.Conv2D( filters=hp.Int('conv_2_filter', min_value=32, max_value=96, step=16), kernel_size=hp.Choice('conv_2_kernel', values = [5,3]), activation='relu', padding = 'same' ), keras.layers.Conv2D( filters=hp.Int('conv_3_filter', min_value=32, max_value=64, step=16), kernel_size=hp.Choice('conv_3_kernel', values = [7,3]), activation='relu', padding = 'same' ), keras.layers.MaxPooling2D( pool_size=hp.Choice('maxpool_1_size', values = [2,2]), ), keras.layers.Flatten(), keras.layers.Dense( units=hp.Int('dense_1_units', min_value=32, max_value=128, step=16), activation='relu' ), keras.layers.Dense(1, activation='relu') ])

model.compile(optimizer=keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3,1e-4])), loss='mean_absolute_error', metrics=[R2_Score])

return model

As I am working with regression I used this fuction for R2Score as in Keras they lack of it:

def R2_Score(y_true, y_pred): from keras import backend as K SS_res = K.sum(K.square( y_true-y_pred )) SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) ) return ( 1 - SS_res/(SS_tot + K.epsilon()) )

For tunning I am using this code:

tuner=RandomSearch(build_model, objective= kerastuner.Objective('R2_Score', direction='max'), max_trials=5, directory='output', project_name="CRISPR-Cas9")

tuner.search(X_train, y_train, epochs=5, validation_data=(X_test, y_test), metrics=[R2_Score])

I get the error in model.compile, and I suspect that I will get the same in the tuner or tuner.search:

INFO:tensorflow:Reloading Oracle from existing project output/CRISPR-Cas9/oracle.json

Search: Running Trial #1

Hyperparameter |Value |Best Value So Far conv_1_filter |64 |? conv_1_kernel |3 |? conv_2_filter |48 |? conv_2_kernel |3 |? conv_3_filter |32 |? conv_3_kernel |7 |? dense_1_units |96 |? learning_rate |0.0001 |?

---

TypeError Traceback (most recent call last)

in () 8 epochs=5, 9 validation_data=(sequences_test, indel_test), ---> 10 metrics=['R2_Score']) 3 frames /usr/local/lib/python3.6/dist-packages/kerastuner/engine/base_tuner.py in search(self, *fit_args, **fit_kwargs) 129 130 self.on_trial_begin(trial) --> 131 self.run_trial(trial, *fit_args, **fit_kwargs) 132 self.on_trial_end(trial) 133 self.on_search_end() /usr/local/lib/python3.6/dist-packages/kerastuner/engine/multi_execution_tuner.py in run_trial(self, trial, *fit_args, **fit_kwargs) 94 copied_fit_kwargs['callbacks'] = callbacks 95 ---> 96 history = self._build_and_fit_model(trial, fit_args, copied_fit_kwargs) 97 for metric, epoch_values in history.history.items(): 98 if self.oracle.objective.direction == 'min': /usr/local/lib/python3.6/dist-packages/kerastuner/engine/tuner.py in _build_and_fit_model(self, trial, fit_args, fit_kwargs) 139 """ 140 model = self.hypermodel.build(trial.hyperparameters) --> 141 return model.fit(*fit_args, **fit_kwargs) 142 143 def run_trial(self, trial, *fit_args, **fit_kwargs): /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs) 106 def _method_wrapper(self, *args, **kwargs): 107 if not self._in_multi_worker_mode(): # pylint: disable=protected-access --> 108 return method(self, *args, **kwargs) 109 110 # Running inside `run_distribute_coordinator` already. TypeError: fit() got an unexpected keyword argument 'metrics' I saw some solutions for this problem but none work form me, also almost every example of CNN I see is used in clasification but I am doing a regression. Can anybody give me a hand? — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or unsubscribe .

[whtop]

@Shivayv That's because you need to supply a direction since it can't be inferred.

Also my mistake metrics needs to be supplied in compile() not fit()

from tensorflow import keras
from tensorflow.keras import layers
from kerastuner.tuners import RandomSearch
import tensorflow as tf
import kerastuner

def build_model(hp):
    model = keras.Sequential()
    model.add(layers.Dense(units=hp.Int('units',
                                        min_value=32,
                                        max_value=512,
                                        step=32),
                           activation='relu'))
    model.add(layers.Dense(1, activation='sigmoid'))
    model.compile(
        optimizer=keras.optimizers.Adam(
            hp.Choice('learning_rate',
                      values=[1e-2, 1e-3, 1e-4])),
        loss='binary_crossentropy',
        metrics=[tf.keras.metrics.AUC()])
    return model

tuner = RandomSearch(
    build_model,
    objective=kerastuner.Objective("auc", direction="min"),
    max_trials=5,
    executions_per_trial=3,
    directory='my_dir',
    project_name='helloworld')

x = [[1], [2], [3]]
y = [[0], [1], [0]]

tuner.search(x, y, epochs=5)

Shouldn't the direction for AUC be "max" rather than min?

[hanneswiedenhofer]

Same problem here. Using Colab. Keras 2.4.3 Tensorflow 2.4.0 Any suggestions?

[haifeng-jin]

Yes, I think the direction should be max.

[PaoloRanzi81]

@whtop I think you are right. For Area Under the Curve (AUC) the direction of the optimization should be "max". E.g. a great classifier with a AUC of 0.9 performs better than a terrible classifier which instead has a AUC of 0.6. In other words, the larger the area the better.

[Algernonthanks]

hi have you solve this problem. If solving, could you please share your solution?