Error execute with new version and Delphi 10.3

[JeanYvesJonet]

Hello, after the problem with TNNet class missing, i have a new problem when I execute a code without problem with old version. Classe d'exception Exception avec un message 'TNNetInput:32;32;1;0;0;0;0;0 not allowed in CreateLayer Thanks

[joaopauloschuler]

This commit should fix: https://github.com/joaopauloschuler/neural-api/commit/31f7b2e81ff651123ac3f1e10040f98fd1c8eb9d

[joaopauloschuler]

@JeanYvesJonet , does this commit fix?

[JeanYvesJonet]

Yeah, it's correct. Thank you very much for you reactivity. Now, I can try the new function TNNetswish. 🥇

[joaopauloschuler]

Glad to help!

[joaopauloschuler]

@JeanYvesJonet , just wondering, how was your TNNetswish trial?

Are you working with convolutional layers?

[JeanYvesJonet]

Hello, I tried it only with ReLU replacement, but the result is not good. Is-it normal that there are no parameter for Netswish ? Perhaps, i don't used it very well into my convolutional layers. Have you a model which use it for pictures (32x32 1D) learning with 2 classes ? Thanks.

[joaopauloschuler]

I would try something like this:

  // Padding and cropping constants.
  csPadding = 4;
  csCropSize = csPadding * 2;

  procedure MonochromeImageClassifier;
  var
    NN: TNNet;
    NeuralFit: TNeuralImageFit;
    ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes: TNNetVolumeList;
  begin
    WriteLn('Creating Neural Network...');
    NN := TNNet.Create();
    NN.AddLayer([
      TNNetInput.Create(32, 32, 1),
      TNNetConvolutionLinear.Create({Features=}32, {FeatureSize=}5, {Padding=}2, {Stride=}1, {SuppressBias=}1),
      TNNetMaxPool.Create(4),
      TNNetMovingStdNormalization.Create(),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1, {SuppressBias=}1),
      TNNetSwish.Create(),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1, {SuppressBias=}1),
      TNNetSwish.Create(),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1, {SuppressBias=}1),
      TNNetSwish.Create(),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1, {SuppressBias=}1),
      TNNetSwish.Create(),
      TNNetDropout.Create(0.5),
      TNNetMaxPool.Create(2),
      TNNetFullConnectLinear.Create(2), // 2 classes
      TNNetSoftMax.Create()
    ]);
    NN.DebugStructure();

    //TODO: load ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes) here.

    // Add padding to dataset
    WriteLn
    (
      'Original image size: ',
      ImgTrainingVolumes[0].SizeX,',',
      ImgTrainingVolumes[0].SizeY,' px.'
    );
    ImgTrainingVolumes.AddPadding(csPadding);
    ImgValidationVolumes.AddPadding(csPadding);
    ImgTestVolumes.AddPadding(csPadding);

    WriteLn
    (
      'New image size after padding: ',
      ImgTrainingVolumes[0].SizeX,',',
      ImgTrainingVolumes[0].SizeY,' px.'
    );

    NeuralFit := TNeuralImageFit.Create;

    // Enable cropping while fitting.
    NeuralFit.HasImgCrop := true;
    NeuralFit.MaxCropSize := csCropSize;

    NeuralFit.FileNameBase := 'SimpleImageClassifierPaddingCroppingSwish-'+IntToStr(GetProcessId());
    NeuralFit.InitialLearningRate := 0.001;
    NeuralFit.LearningRateDecay := 0.01;
    NeuralFit.StaircaseEpochs := 10;
    NeuralFit.Inertia := 0.9;
    NeuralFit.L2Decay := 0.00001;

    NeuralFit.Fit(NN, ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, {NumClasses=}10, {batchsize=}64, {epochs=}50);
    NeuralFit.Free;

    NN.Free;
    ImgTestVolumes.Free;
    ImgValidationVolumes.Free;
    ImgTrainingVolumes.Free;
    Terminate;
  end;

[joaopauloschuler]

Observe that the first layer has no activation function and is followed by a maxpool:

TNNetConvolutionLinear.Create({Features=}32, {FeatureSize=}5, {Padding=}2, {Stride=}1, {SuppressBias=}1),
TNNetMaxPool.Create(4),

The last layers are a fully connected layer without activation followed by a softmax:

TNNetFullConnectLinear.Create(2), // 2 classes
TNNetSoftMax.Create()

For the input data, try to scale it from -2 to +2 instead from 0 to 255.

[JeanYvesJonet]

Thank you very much, i will try it and send you a feedback when the training and test will be terminate. I understand my problem with swish, i forgot to add a convolution linear before.

[JeanYvesJonet]

Hello Joao-Paulo, i tried with my samples (scale -1/+1 for input data), train with 5183 images class 0 and 18935 images class 1. Test samples : 208 639 images class 0 and 475 763 class 1.
Training time, 3:12 after 130 epochs for swish, 6:27 after 230 epochs for ReLU. (until error=0 for TRAIN images) Result with TEST images : 36 errors with swish (13 class 0 and 23 class 1) and 52 with ReLU (39 class 0 and 13 class 1) 0.52/10 000 errors for swish and 0.76/10 000 errors for ReLU. Decrease of 30% the errors for this try. I will try with scale -2/+2 soon as possible. Thanks.

[JeanYvesJonet]

Sorry, the scale of inputdata as you suggest is-it for TNNetVolume of image input or TNNetVolume for output ? Currently i transforme image to 0-1 values (instead of 0-255 gray level), and i write output with 0 / +1 scale. Thanks

[joaopauloschuler]

The suggestion was for input. For classification tasks, I usually use a softmax as last layer and expect output as [0, +1]. A fully implemented example can be found here: https://github.com/joaopauloschuler/neural-api/blob/master/examples/SimpleImageClassifier/SimpleImageClassifierPaddingCroppingSwish.lpr .