Proposed enhancements

[libornovax]

Try these:

• Leaky ReLu ✅ ⛔️
• Increase the number of parameters in the layers (my net has only 9MB in total as opposed to SSD, which has 100MB or YOLO9000, which has 224MB!) ✅ 🆗
• Increase the number of conv layers to increase the expressiveness of the network ✅ 🆗
• Use weights from VGGnet ❌

[libornovax]

Increasing the number of parameters

As shown below, increasing (approximately doubling) the number of parameters improved the performance of the network, when the same learning setup was used.

Original network More parameters More parameters, uniform lr = 0.0005

Note: We can see stagnation after 5000 training iterations. That is because at that point we decreased the learning rate by the factor of 10, which was not a good idea as the gradient is already pretty small. That is why I ran one more test with uniform learning rate, which seems to be better.

[libornovax]

Uniform and higher learning rate

Using uniform learning rate instead of decreasing it showed to be better! Also higher learning rate was favorable.

lr = 0.0001, decreasing every 5000 iterations lr = 0.0001, fixed lr = 0.0005, fixed

[libornovax]

More convolutional layers

I changed the network to have more convolutional layers, but the same field of view. Here is the model:

macc_0.25_r2_x4
r2 c0.25
conv k3      o64
conv k3      o64
pool
conv k3      o128
conv k3  d2  o128
conv k3      o128
pool
conv k3      o256
conv k3  d1  o256
conv k3  d3  o256
conv k3  d5  o256
conv k3      o256
macc x4

Original network, decreasing learning rate More convolutional layers, decreasing learning rate More convolutional layers, fixed lr = 0.0005

It seems that adding more convolutional layers helps, but apparently we almost reached some bottom since the higher learning rate does not improve the performance too much.

[libornovax]

Comparison of training with lr = 0.0005

I show 4 plots. All of those trainings are ran with the same setup lr = 0.0005, 20000 iterations, batch size 32. KITTI train and Jura test short as validation (as in all tests). The plots compare the influence of introduced changes to the network. Also, I made one more run with using leaky ReLu units.

Leaky ReLu

Note: We see that the validation error is basically the same in all trainings. However introducing more conv layers or more parameters increases the networks' ability to fit to the training data. As can be seen, the more conv and more param networks "fit" more to the training data - they have a lower loss on the training set.

It also seems that we have reached some minimum value of loss on the validation set as we are not able to decrease the error by doing any of the changes that we introduced.