Early Stop Implementation

[NEGU93]

Early Stop Implementation. The implementation is changed to the following:

backpropagation(
    # Required parameters
    network,                     # the neural network instance to train
    training_set,                # the training dataset
    test_set,                    # the test dataset
    cost_function,               # the cost function to optimize
    # Optional parameters
    ERROR_LIMIT=1e-3,                           # Error tolerance when terminating the learning
    max_iterations=(),                          # Regardless of the achieved error, terminate after max_iterations epochs. Default: infinite
    batch_size=0,                               # Set the batch size. 0 implies using the entire training_set as a batch, 1 equals no batch learning, and any other number dictate the batch size
    input_layer_dropout=input_layer_dropout,    # Dropout fraction of the input layer
    hidden_layer_dropout=hidden_layer_dropout,   # Dropout fraction of in the hidden layer(s)
    print_rate=1000,                            # The epoch interval to print progression statistics
    save_trained_network=False,                 # Whether to ask the user if they would like to save the network after training
    early_stop=()
)

Where early_stop = () would be not to do early stop. Otherwise the number must be an integer, which will mean the number of epochs as tolerance, it means that if the tendency of the Error of the test set has increased for more than the giving tolerance epochs, then stop the training.

I have examples and plots and analysis on the optimal value for early stop should you want to add some to the documentation page.

Posible Issues and TODO things:

1. The algorithm writes a variable in each epoch and performs a comparison even if the early stop is not required... It may affect (very little impact anyway) the performance.
2. The algorithm prints print "[training] Converged to error bound (%.4g) with error %.4g." % ( ERROR_LIMIT, error ) which I think will not be the case if early stop was applied.

[wizzwizz4]

Why () and not None? Is it to match with other parameters?

[NEGU93]

Sorry my delay, it was a busy day.

I tried to copy the previous structure... the data goes as the max_iterations parameter. It's just a "no max iterations will be a max_iterations = ()" and then an unsigned int... (same as the early_stop idea)

But that's the only reason... Personally, under other circumstances, I'd like the None idea better but I thought to make it similar to the existing code.

So to conclude, yes, it is as you say, to match with other parameters.