Equivalent to Dropout - Githubissues

Dzandaa commented 1 year ago

Hello,

Is there and equivalent to "dropout" in CAI?

Thank you.

B->

joaopauloschuler commented 1 year ago

Hello!

Have you tried TNNetDropout.Create(Rate: double; OneMaskPerbatch: integer = 1); ?

Glad to help.

Dzandaa commented 1 year ago

Hi, Thank you, I missed it :)

Rate is from 0 to 100, right?

B->

Dzandaa commented 1 year ago

Hi again,

I have a directory with 8 sub-directories containing images (64x64) of faces expressions.

I try to train this network:

`
VGG13NN := TNNet.Create; // VGG13 Custom Net VGG13NN.AddLayer([ TNNetInput.Create(64, 64, 3),

TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetMaxPool.Create(2), // TNNetDropout.Create(25, 1),

TNNetConvolutionReLU.Create({Features=}128, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetConvolutionReLU.Create({Features=}128, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetMaxPool.Create(2), // TNNetDropout.Create(25, 1),

TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1), TNNetMaxPool.Create(2), // TNNetDropout.Create(25, 1),

TNNetFullConnectReLU.Create(1024), // TNNetDropout.Create(50, 1), TNNetFullConnectReLU.Create(1024), // TNNetDropout.Create(50, 1),

TNNetLayerFullConnect.Create(8), TNNetSoftMax.Create() ]);

VGG13NN.DebugStructure(); VGG13NN.SetLearningRate(0.001,0.9); VGG13NN.SetL2Decay(0.0);
`
I load the images in volumes with this code:

` LoadOK := False; if(SelectPicturesDirectory.Execute) then begin // change ProportionToLoad to a smaller number if you don't have available 16GB of RAM. ProportionToLoad := 0.1;

MLog.Append('Loading ' + Round(ProportionToLoad*100).ToString + ' % of The Faces Dataset into memory.');
CreateVolumesFromImagesFromFolder
(
 ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes,
 {FolderName=}SelectPicturesDirectory.FileName, {pImageSubFolder=}'',
 {color_encoding=}0{RGB},
 {TrainingProp=}0.9*ProportionToLoad,
 {ValidationProp=}0.05*ProportionToLoad,
 {TestProp=}0.05*ProportionToLoad,
 {NewSizeX=}64, {NewSizeY=}64
);

MLog.Append('Training Images: ' +  ImgTrainingVolumes.Count.ToString);
MLog.Append('Validation Images: ' +  ImgValidationVolumes.Count.ToString);
MLog.Append('Test Images: ' +  ImgTestVolumes.Count.ToString);

LoadOK := True;

end;
` It seems O.K.

And I train with this code:

` VGG13Fit := TNeuralImageFit.Create(); MLog.Append('Computing...');

VGG13Fit.OnAfterEpoch := @Self.OnAfterEpoch; VGG13Fit.OnAfterStep := @Self.OnAfterStep; VGG13Fit.OnStart := @Self.OnStart; VGG13Fit.FileNameBase := 'ExpressionDetection'; VGG13Fit.InitialLearningRate := 0.001; VGG13Fit.LearningRateDecay := 0.01; VGG13Fit.StaircaseEpochs := 10; VGG13Fit.Inertia := 0.9; VGG13Fit.L2Decay := 0.00001;

if HasOpenCL then begin VGG13Fit.EnableOpenCL(EasyOpenCL.PlatformIds[0], EasyOpenCL.Devices[0]); VGG13NN.EnableOpenCL(EasyOpenCL.PlatformIds[0], EasyOpenCL.Devices[0]); end; IsFitting := True;

VGG13Fit.Fit(VGG13NN, ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, {NumClasses=}8, {batchsize=}SEBatch.Value, {epochs=}SEEpoch.Value);

IsFitting := False; MLog.Append('Done');
`

Nothing happens, Never go to OnStart, OnAfterStep, OnAfterEpoch But the program use more and more memory.

The test use 1332 images.

Any Idea?

Thank you.

B->

joaopauloschuler commented 1 year ago

Just for debugging, try to add VGG13Fit.MaxThreadNum := 1; before calling VGG13Fit and check memory usage. You'll have one full memory structure for each parallel thread.

joaopauloschuler commented 1 year ago

I would also recommend giving a try to this model:

      NN.AddLayer([
      TNNetInput.Create(64, 64, 3),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}5, {Padding=}4, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetMovingStdNormalization.Create(),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}2),
      TNNetDropout.Create(0.5),
      TNNetMaxPool.Create(2),
      TNNetFullConnectLinear.Create(8),
      TNNetSoftMax.Create()
    ]);

This a model that I used in a number of experiments at: https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

joaopauloschuler commented 1 year ago

You can increase MaxThreadNum bit by bit checking memory usage, if you like the idea.

Dzandaa commented 1 year ago

VGG13 Net.pdf

This is the model I tried to reproduce.

B->

joaopauloschuler commented 1 year ago

@Dzandaa , I'm curious if you had a chance to test with smaller number of threads and / or the other simpler model above.

Dzandaa commented 1 year ago

Hi,

Still testing :)

When you convert RGB images to TNeuralFloat, what is the range of NeuralFloat?

RGB = 0..255 NeuralFloat =?

Thank you.

B->

Dzandaa commented 1 year ago

Sorry, I didn't see that there is a NeuronalInputToRgbImg function :)

I've go some good results using your model.

I think mine is to big for my computer and graphic card.

To be continued. Expression

B->

joaopauloschuler commented 1 year ago

@Dzandaa , absolute many thanks for sharing this beautiful screenshot!!!

In the case that you haven't looked at, I highly recommend having a look at: https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

In the case that you would like to try a more complex model, you could start with:

      TNNetInput.Create(64, 64, 3),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}5, {Padding=}4, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetMovingStdNormalization.Create(),

Then, you could continue with a "RESNET" style: https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/CaiResNet20.lpr https://github.com/joaopauloschuler/neural-api/tree/master/examples/ResNet

For using the trained NN in production, you can follow the example from (if you like): https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

It would be:

  procedure ClassifyOneImageSimple;
  var
    NN: TNNet;
    ImageFileName: string;
    NeuralFit: TNeuralImageFit;
  begin
    WriteLn('Loading Neural Network...');
    NN := TNNet.Create;
    NN.LoadFromFile('SimplePlantLeafDisease-20230720.nn');
    NeuralFit := TNeuralImageFit.Create;
    ImageFileName := 'plant/Apple___Black_rot/image (1).JPG';
    WriteLn('Processing image: ', ImageFileName);
    WriteLn(
      'The class of the image is: ',
      NeuralFit.ClassifyImageFromFile(NN, ImageFileName)
    );
    NeuralFit.Free;
    NN.Free;
  end;

Dzandaa commented 1 year ago

Hi, Thank you for the links.

I've already tested "Plant leaf Disease" and "Colorectal" NetworkTest

What I try to do with CAI, is to detect Atoms in a Spectrum

The input is a TSpectrum of 1024 values

type TSpectrum = array[0..1023] of TNeuralFloat;

And the output is the detected atom(s) In this case I try to identify only 5 Atoms.

this is the model:

`NeuralNet.AddLayer([

TNNetInput.Create(1024),

TNNetConvolutionReLU.Create(32, 3, 1, 1, 1),

TNNetMaxPool.Create(2, 2, 0),

TNNetConvolutionReLU.Create(32, 3, 1, 1, 1),

TNNetMaxPool.Create(2, 2, 0),

TNNetConvolutionReLU.Create(5, 3, 1, 1, 1),

TNNetFullConnectLinear.Create(5)

]); `

Spectrum

Of course the Spectrum is just a simulation with some peaks :)

B->

joaopauloschuler / neural-api

Equivalent to Dropout #117