Feature request: Early stopping with patience

[renatobellotti]

I would like to have patience in the early stopping code.

As I understood it we currently stop once the loss value drops by less than delta compared to the last iteration. I suggest to stop if the loss has not decreased by delta in the last n iterations, where n is the patience. This technique is very common in machine learning, see e. g. https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping.

[pkofod]

That's possible, yes. Requires a bit of bookkeeping.

[renatobellotti]

I have implemented it now as a callback. I hope I get around to create a merge request at some point.

"""
Stops the iteration if the function value has not decreased by more than delta in the last patience iterations.
"""
function build_early_stopping(delta::T, patience, store_trace=false) where {T}
    previous_best = typemax(T)
    previous_best_iteration = 1

    function early_stopping(optimizationState)
        if optimizationState.value <= (previous_best - delta)
            previous_best = optimizationState.value
            previous_best_iteration = optimizationState.iteration
        end

        return (optimizationState.iteration - previous_best_iteration) > patience
    end

    function early_stopping_with_trace(trace)
        return early_stopping(trace[end])
    end

    if store_trace
        early_stopping_with_trace
    else
        return early_stopping
    end
end

The case distinction is needed to distinguish the choices for store_trace in the optimisation options.