Allow to stop after max time reached

[letmaik]

It would be convenient to be able to specify a maximum time after which no more trials should be tried. Using max_trials is sometimes hard to get right if what you actually want is to run as many trials as possible in a given time frame (your budget). I couldn't find an easy way to do that. Ray's tune has Stoppers which make this quite easy.

[Erik-BM]

Is this what you are looking for https://www.tensorflow.org/addons/tutorials/time_stopping ? Let's you at least set a max time for each trial, which is not a problem if the trials take similar amount of time.

[letmaik]

No, I don't mean per-trial, but overall.

[ben-arnao]

Assuming you want to track the total time elapsed of the tuner when it is running (ie. if you stop and load the tuner, we want to account for this) for a crude example we could do something like this...

basetuner.py

def search(self, *fit_args, **fit_kwargs):
    """Performs a search for best hyperparameter configuations.
    # Arguments:
        *fit_args: Positional arguments that should be passed to
          `run_trial`, for example the training and validation data.
        *fit_kwargs: Keyword arguments that should be passed to
          `run_trial`, for example the training and validation data.
    """
    self.on_search_begin()
    import time
    MAX_TIME_ALLOWED = 1000
    while True:
        trial_start = time.time()
        trial = self.oracle.create_trial(self.tuner_id)
        if trial.status == trial_module.TrialStatus.STOPPED:
            # Oracle triggered exit.
            tf.get_logger().info('Oracle triggered exit')
            break
        if trial.status == trial_module.TrialStatus.IDLE:
            # Oracle is calculating, resend request.
            continue

        self.on_trial_begin(trial)
        self.run_trial(trial, *fit_args, **fit_kwargs)
        self.on_trial_end(trial)
        trial.update('elapsed', time.time() - trial_start)
        if np.sum(trial.metrics.get_history('elapsed')) > MAX_TIME_ALLOWED:
            break
    self.on_search_end()

[letmaik]

Yes, it would be great if this was integrated somehow in Keras Tuner to avoid people duplicating the above code.

[MbProg]

Running the code above, I get an error message 'Trial' object has no attribute 'update'. It seems that trial.metrics.update works. But apart from this, it seems to me that the mentioned code just adds the elapsed time to each trial which is overwritten by the next trial object created by trial = self.oracle.create_trial(self.tuner_id) in the next iteration. So why not just saving the starting time in a class member instead of using trial.update?