Bug or miss configuration by a newbie ? (fixed)

[HuguesDug]

Hello,

First, thanks for this hard work you did. Promising project.

Development environment is Delphi 10.3, Master branch with no modification as of June 21rst.

I wanted to program a simple NN in a windows form environment. As most of the debug possibilities of the library are sent to console with WriteLn, I copied the concept of "Or Xor And" example, with no NeuralFit.

Then, I started with the simplest possible network. One layer, no bias, fully connected. Input and output data are identity function --> VOuptput[i]= Vinput[i] Vinput are in a range 0 to 1, 500 values.

So, 1 neuron linking input to output. Its value should converge to 1 during Training.

Network creation; NN := TNNet.Create(); NN.AddLayer(TNNetInput.Create(1)); NN.AddLayer(TNNetFullConnect.Create(1, 1)); NN.InitWeights;

Training procedure: var Cnt, EpochCnt: integer; begin pOutPut := TNNetVolume.Create(1, 1);

NN.SetLearningRate(0.01, 0.9); for EpochCnt := 1 to 3000 do begin for Cnt := Low(vInputs) to High(vInputs) do begin NN.Compute(vInputs[Cnt]); NN.GetOutput(pOutPut); NN.Backpropagate(vOutput[Cnt]); end; end; pOutPut.Free;

The result of the training is very strange. Weight starts with a value of 2.68, then increase at each "backprop", instead of decreasing to 1. After 3000 loops of training, it is stuck at a value of 3.22 and no longer change.

I also tested with the NeuralFit process, similar behavior. It runs with no CPU consumption after a short while, and never returns as finished.

I also added more layer, reactivated the bias, ... do difference.. System is not converging. The biggest suprise is that it stops to evoluate, even if in the wrong direction.

I will copy my code in Lazarus (latest release) and look if it does the same.

[HuguesDug]

I tested in Lazarus and had quite a hard time to get it going. In particular, a Threading unit not installed and not willing to install. Same behavior. Then, I debugged the code, line by line... and discovered a fullconneconnected layer has in fact a "Tanh" activation function... Having changed to fullyconnectedlinear gives the expected result. Some comments in the code of the activation functionned in use would be appreciated.

[joaopauloschuler]

Thank you for the interesting report. The easiest and most stable activation function is ReLU. So, I would recommend starting from layers that have ReLU.

About threading, in Lazarus, menu Project, Project Inspector, Required Packages, please add multithreadprocslaz.

Have you tried cai/lazarus/examples/XorAndOr/XorAndOr.lpi Lazarus project?

[joaopauloschuler]

BTW, thank you for sharing insights so I can improve the documentation!

[HuguesDug]

Hello,

I tryed some other libraries like FAAN some years ago. ReLu has some advantages, but for some networks with learning values negative or very negative, it can sometime be creating some issues. I just wanted to test the lib at this stage before going "bigger".

For Lazarus,, I found the multithreadprocslaz.... But this would require a comment at the declaration, so users know that this is the package to install.

In general, more documentation (and comments) in the code to describe the functions would be a big step forward. At this stage, you have to read the code to understand what it does;

XorAndOr. lpi. I tested the one with NeuralFit, and the one without (and most of the examples). As said, I want to see the evolution of the learning/testing process inside a windows form application. Debug of Neural fit goes only into console (a patch to call a "threadsafe customizable display function" would be great. So, I went for the no NeuralFit version.

Anyway, great job.

I found CAI library some years ago onto Sourceforge. At that time it was lazarus only. I posted a request for modification, seeing that translation was possible. Glad that somebody did it.

[joaopauloschuler]

About NeuralFit and "debug of neural fit goes only into console", if you need to hide messages, you can call:

      procedure HideMessages();

You can override message treatment with:

      property MessageProc: TGetStrProc read FMessageProc write FMessageProc;
      property ErrorProc: TGetStrProc read FErrorProc write FErrorProc;

NeuralFit descends from TMObject that allows you to code your own message treatment:

  /// Class with string message events
  TMObject = class(TObject)
    protected
      FMessageProc: TGetStrProc;
      FErrorProc: TGetStrProc;

    public
      constructor Create();
      destructor Destroy(); override;

      procedure DefaultMessageProc(const S: string);
      procedure DefaultErrorProc(const S: string);
      procedure DefaultHideMessages(const S: string);
      procedure HideMessages();

    published
      property MessageProc: TGetStrProc read FMessageProc write FMessageProc;
      property ErrorProc: TGetStrProc read FErrorProc write FErrorProc;
  end;

On your own code, you could do is:

  MyFit.MessageProc := {$IFDEF FPC}@{$ENDIF}Self.MessageProc;
  MyFit.ErrorProc := {$IFDEF FPC}@{$ENDIF}Self.ErrorProc;

[HuguesDug]

Hello, I will give this a try.

You have still some "writeln" in the code in many places. However, they seem to be covered by the verbose condition. So, with no verbose, should work.