neural network training

[dadrake3]

Is there a way to terminate the training of a network if the error rate has converged to a certain significant figure?

[mc10011]

This sounds similar to the problem described in this link:

https://www.researchgate.net/post/Why_is_Pyrren_in_python_taking_forever_to_run

I have encountered this problem as well and added a "minimum error step criterium" in Train_LM function to alleviate it. The added criteria tackles the symptom, i.e. training running forever, and your request. My solution gives room for five more iterations until it breaks the training once and for all.

See my modified train_LM below. You can set the early stopping criteria as 1.0 instead of 5.0 and use the min_E_step as a convergence criteria for early stopping the training.

`def train_LM(P,Y,net,k_max=100,E_stop=1e-10,dampfac=3.0,dampconst=10.0,\ verbose = False,min_E_step=1e-09): """ Implementation of the Levenberg-Marquardt-Algorithm (LM) based on: Levenberg, K.: A Method for the Solution of Certain Problems in Least Squares. Quarterly of Applied Mathematics, 2:164-168, 1944. and Marquardt, D.: An Algorithm for Least-Squares Estimation of Nonlinear Parameters. SIAM Journal, 11:431-441, 1963.

Args:

    P:      NN Inputs
    Y:      NN Targets
    net:    neural network
    k_max:  maxiumum number of iterations
    E_stop: Termination Error, Training stops when the Error <= E_stop
    dampconst:  constant to adapt damping factor of LM
    dampfac:    damping factor of LM
    min_E_step: minimum step for error. When reached 5 times, training terminates.        
Returns:
    net:    trained Neural Network 
"""
#create data dict
data,net = prepare_data(P,Y,net)

#Calculate Jacobian, Error and error vector for first iteration
J,E,e = RTRL(net,data)
k = 0
ErrorHistory=np.zeros(k_max+1) #Vektor for Error hostory
ErrorHistory[k]=E
if verbose:
    print('Iteration: ',k,'     Error: ',E,'    scale factor: ',dampfac)

early=0

while True:
#run loop until either k_max or E_stop is reached

    JJ = np.dot(J.transpose(),J) #J.transp * J
    w = net['w'] #weight vector
    while True:
    #repeat until optimizing step is successful
        #gradient
        g = np.dot(J.transpose(),e)

        #calculate scaled inverse hessian
        try:
            G = np.linalg.inv(JJ+dampfac*np.eye(net['N'])) #scaled inverse hessian
        except np.linalg.LinAlgError:
            # Not invertible. Go small step in gradient direction
            w_delta = 1.0/1e10 * g
        else:
            # calculate weight modification
            w_delta = np.dot(-G,g)

        net['w'] = w + w_delta #new weight vector

        Enew = calc_error(net,data) #calculate new Error E          
        if Enew<E and abs(E-Enew)>=min_E_step:
        #Optimization Step successful!
            #if E-Enew<=1e-09:
            dampfac= dampfac/dampconst#adapt scale factor
            break #go to next iteration
        else:
        #Optimization Step NOT successful!\
            dampfac = dampfac*dampconst#adapt scale factor
            if abs(E-Enew)<=min_E_step:
                if verbose:
                    print('E-Enew<=min_E_step Encountered!!')
                early=early+1
                if early>=5.0:
                    if verbose:
                        print('5 Times * E-Enew<=min_E_step Encountered!!')
                break                    

    #Calculate Jacobian, Error and error vector for next iteration
    J,E,e = RTRL(net,data)
    k = k+1
    ErrorHistory[k] = E
    if verbose:
        print('Iteration: ',k,'     Error: ',E,'    scale factor: ',dampfac)

    #Ceck if termination condition is fulfilled
    if k>=k_max:
        print('Maximum number of iterations reached')
        break
    elif E<=E_stop:
        print('Termination Error reached')
        break
    elif early>=5.0:
        print('Error decreased 5 times by minimum step. Force training exit.')
        break

net['ErrorHistory'] = ErrorHistory[:k]
return net`