Add support for convolutional, pooling and flatten layer

[free43]

Added implementation of forward and backward propagation for following layers:

• ConvolutionalLayer
• PoolingLayer
• FlattenLayer

For _calculate_hiddendelta calculation is forwarded to the responsible class, so no repeated implementation.

The used format for input data and their delta is: NxCHxCxR (Batch samples x Channels x Columns x Rows). For kernels the format CHxCxRxK is used. This can be seen in the file 4d_dimensions.h.

Added unit and integration test cases. System test is done via the example mnist. Here 70,000 input data is used. Following time measurements are obtained during training on the system OS: Ubuntu 22, Processor: Intel® Core™ i3-2330M CPU @ 2.20GHz × 4, Memory: 8,0 GiB:

• Model 1 without conv layers: 16 s
• Model 2 1xMaxPoolingLayer and 2xconv layers: 36 min 1 s
• Model 3 LeNet5: 30 min 3 s

Further details can be seen in the file mnist example. Also the confusion matrix is printed there.

Next steps: performance improvement, adapt other classes like TestingAnalysis to support 3/4D input dimension, add comments and checks

PS: Changed the order of assemble gradient and applying regularization to the weights (commit). The reason is, assemble gradient will override the applied regularization from the previous batch.

[leondavi]

@free43 Is this change adding support in de-convolution as well?

[free43]

@free43 Is this change adding support in de-convolution as well?

The transposed convolution is implemented, which is often named "deconvolution". See the method ConvolutionalLayer::calculate_hidden_delta. The math behind of this can be read here cnn_backpropagation_article.pdf which was uploaded originally here https://codeandfire.github.io/assets/2021-07-16-backpropagation/article.pdf. Last access 12.10.23. The described method can be used to derive a "deconvolution" with multiple kernels and channels.