Bug: overflow-error in Windows with 64-bit-fannfloat.dll

[Jens01]

I had a overflow-error with the 64bit fannfloat.dll. rprop_increase_factor and rprop_decrease_factor were similar to 0 Fixed it with this issue from ache7: https://github.com/libfann/fann/issues/85

fann_train.c

void fann_update_weights_irpropm(struct fann *ann, unsigned int first_weight, unsigned int past_end)
{
    fann_type *train_slopes = ann->train_slopes;
    fann_type *weights = ann->weights;
    fann_type *prev_steps = ann->prev_steps;
    fann_type *prev_train_slopes = ann->prev_train_slopes;

    double prev_step, next_step, slope, prev_slope, same_sign;

    float increase_factor = ann->rprop_increase_factor; /*1.2; */
    float decrease_factor = ann->rprop_decrease_factor; /*0.5; */
    float delta_min = ann->rprop_delta_min; /*0.0; */
    float delta_max = ann->rprop_delta_max; /*50.0; */

    unsigned int i = first_weight;

    for(; i != past_end; i++)
    {

        prev_step = fann_max(prev_steps[i], (fann_type) 0.0001);    /* prev_step may not be zero because then the training will stop */

        slope = train_slopes[i];
        prev_slope = prev_train_slopes[i];

        same_sign = prev_slope * slope;

// https://github.com/libfann/fann/issues/85
        next_step = prev_step;
        if (same_sign > 0)
            next_step = fann_min(prev_step * increase_factor, delta_max);
        else if (same_sign < 0) {
            next_step = fann_max(prev_step * decrease_factor, delta_min);
            slope = 0;
        }

        if (slope < 0) {
            weights[i] -= next_step;
            if (weights[i] < -1500) weights[i] = -1500;
        }
        else if (slope > 0) {
            weights[i] += next_step;
            if (weights[i] > 1500) weights[i] = 1500;
        }

    /*
        if(same_sign >= 0.0)
            next_step = fann_min(prev_step * increase_factor, delta_max);
        else
        {
            next_step = fann_max(prev_step * decrease_factor, delta_min);
            slope = 0;
        }

        if(slope < 0)
        {
            weights[i] -= next_step;
            if(weights[i] < -1500)
                weights[i] = -1500;
        }
        else
        {
            weights[i] += next_step;
            if(weights[i] > 1500)
                weights[i] = 1500;
        }
    */

        /*if(i == 2){
         * printf("weight=%f, slope=%f, next_step=%f, prev_step=%f\n", weights[i], slope, next_step, prev_step);
         * } */

        /* update global data arrays */
        prev_steps[i] = next_step;
        prev_train_slopes[i] = slope;
        train_slopes[i] = 0.0;

    }
}

[andersfylling]

@troiganto did you update the code when you closed the previous issue?

[andersfylling]

Thanks for the issue btw. Could you also provide the code to replicate the issue? We would be happy if we could add it as a unit test for future changes/upgrades ^^,

[Jens01]

https://github.com/Jens01/NewFann/blob/cd8cce135b06cbd3d77d38fab10db5edfb2a70ae/FannSource/src/fann_train.c#L722 This is my project : https://github.com/Jens01/NewFann

The error comes during training (not always) when the desired error was very small 0.000001. After the exception rprop_increase_factor and rprop_decrease_factor were similar to 0.

I tested it with the example parity8, cascade and then train in Delphi 64-bit compiler:

  try
    CTrain := TCascadeclass.Create(Fann);
    try
      CTrain.epochs_between_reports := 50;
      Fann.DefNeutronLayer(1, 0.75, FANN_SIGMOID_SYMMETRIC_STEPWISE);

      CTrain.desired_error          := 0.000001;
      CTrain.FannCascadeEvent       := TrainCasEvent;
      CTrain.training_algorithm     := FANN_TRAIN_RPROP;
      CTrain.train_error_function   := FANN_ERRORFUNC_LINEAR;
      CTrain.bit_fail_limit         := 0.5;
      CTrain.train_stop_function    := FANN_STOPFUNC_MSE;
      AcFunctions                   := [FANN_SIGMOID_STEPWISE];
      Steepness                     := [0.5, 1];
      CTrain.activation_functions   := AcFunctions;
      CTrain.activation_steepnesses := Steepness;

      if (CTrain.training_algorithm <> FANN_TRAIN_QUICKPROP) then
      begin
        CTrain.learning_rate    := 0.7;
        CTrain.WeigthsMax       := 5;
        CTrain.WeigthsMin       := -5;
        CTrain.UseRandomWeigths := False;
      end;
      CTrain.TrainCascadeFromFannFile(FNParityTrain);
    finally
      CTrain.Free;
    end;

    for i := 0 to Fann.LayerCount - 1 do
      mmoEvent.Lines.Add(Format('LayerIndx %d : Neurons : %d (%d)', [i, Fann.NeuronCount[i], Fann.NeuronandBiasCount[i]]));

    FannStream.Clear;
    Fann.SaveToStream(FannStream);
    FillNeuronsToList;
  finally
    Fann.Free;
  end;

P.S: Sometimes Github is not my friend...

[troiganto]

@andersfylling I have not. I mis-interpreted it as a misunderstanding about which variant of RProp FANN has implemented. However, I compared the FANN implementation with the original description of iRProp– and there is indeed a difference. I'll see to writing a PR tomorrow.

Writing a unit test about this might, however, be a bit difficult. If I'm understanding Jens01 correctly, they encountered the issue when running a Delphi version of cascade_train.c. I've tried this myself, but couldn't manage to reproduce it. It might be caused by differences in the parameter values, but I haven't been able to narrow it down so far.

[Jens01]

when running a Delphi version of cascade_train.c

First I used cascade after that I trained it. And during the training come the exception with a small desired error and 64bit. I found the error-position with try except in the C++ code.