model_keras Résultats actualisés

[Rob174]

All the code can be found on the enetv1 branch

[Rob174]

Fonction d'approximation des prédiction

The network outputs a vector with the number of element for each class as floats However a non integer number of object is not real. That is why we can use the integer part (noted int) or the round function (noted round) to obtain integers.

Training error results

Valid error results

Interesting elements

Increasing validation error with round

Confusion matrices

🔖 We cannot generate a confusion matrix without approximation function : we would have many different number of objects too often. It would produce a too big confusion matrix

Confusion matrices

👍 Class above 90% accuracy : animal, child,construction_worker,personal_mobility, police_officer, stroller, wheelchair, debris, pushable_pullable, bicycle_rack, bicycle, bus.bendy, bus.rigid, vehicle.construction, ambulance,police, emergency.police, motorcycle

👎 Class below 90% accuracy (generally even below 50%) : 2 subcategories 1️⃣ High variability confusion matrices : adult, barrier, trafficicon, trailer, truck

Common characteristics of confusion matrices

• Wide range of values
• Far more images with a small number of objects (0 or 1)

Consequences

• high variability of the computed precision (order of 10% percents)
• the network often focus itself on the same prediction for a real given number of object. That is why it has a very low accuracy in some case and a relatively good accuracy in some other executions.

2️⃣ Confusion matrices with a better uniform repartition of the number of object (car class only)

Common characteristics

• Wide range of values
• More uniform number of object

Consequences

• More stable accuracy especially for SGD with a 10% range variation for this optimizer
• The network does not focus only on few classes

==> We will have to be careful with high variability

[Rob174]

Impact of number of modules

Training error results

Validation error results

:notebook: Not much improvements from the reference model : we can observe a improvements with 2 modules on the following classes :

• trafficicon : 58% versus 24% (best=86% with 3 modules)
• car : 17% versus 15%
• trailer : 90% versus 48%
• truck : 62% versus 33%

But worse results on the following classes

• adult : 16% versus 26%
• barrier : 16% versus 35% (meilleur : 78% with 3 modules)

[Rob174]

Regularization

Training error %

Validation error %

Confusion matrices

Object type	Reference model error	Best
adult	8%	Regularization everywhere : 63%
barrier	3%	Regularization everywhere : 10%
trafficicon	81%	Regularization everywhere : 87%
trailer	27%	Regularization on only the last dense layer : 77%
truck	56%	Reference model
car	7%	Regularization everywhere : 15%

📓 Best : Regularization everywhere

[Rob174]

Vector transformation layer

Training error %

Validation error %

Note : The GlobalAveragePooling layer is used in the reference model

Confusion matrices

Object type	Reference model error	Best
adult	8%	SPP : 40%
barrier	3%	SPP : 5%
trafficicon	81%	Flatten : 85%
trailer	27%	SPP : 66%
truck	56%	Modèle de référence
car	7%	SPP : 15%

📓 Best : SPP

[Rob174]

Dropout

Training errors metrics

Validation errors metrics

Matrices

Object type	Reference model error	Best
adult	8%	Reference model
barrier	3%	8%, 20% dropout
trafficicon	81%	Reference model
trailer	79%	89%, 20% dropout
truck	56%	Reference model
car	7%	17%, 20% dropout

📓 Best : 50% or 20% dropout

[Rob174]

Final activation function

Training error

Validation error

Confusion matrices precisions

Object type	Reference model error	Best
adult	8%	52% relu
barrier	3%	84% relu
trafficicon	80.61%	86.53% relu or softplus
trailer	79%	90.04% relu
truck	56%	63.28% exp
car	7%	21.27% softplus

📓 Best : relu or softplus

[Rob174]

Adam optimizer : lr and epsilon

Training error

Validation error

Confusion matrices

Object type	Reference model error	Best
adult	8%	57.44% lr=1e-3 ; ε=1e-2
barrier	3%	58.57% lr=1e-5 ; ε=1e-2
trafficicon	80.61%	86% (lr=1e-5 ; ε=1e-2) or (lr=1e-1 ; ε=1e-7) ou (84%) (lr=1e-4 ; ε=1e-7)
trailer	79%	90% (lr=1e-3 ; ε=1e-5)
truck	56%	57% (lr=1e-3 ; ε=1e-5) or reference (lr=1e-3 ; ε=1e-7)
car	7%	20% (lr=1e-3 ; ε=1e-5)

📓 No real best parameters

[Rob174]

Input image size

Training curve

Validation curve

Object type	Reference model error	Best
adult	8%	92% 128 x 72 px
barrier	3%	93% 128 x 72 px
trafficicon	80.61%	97% 128 x 72 px
trailer	79%	94% 128 x 72 px
truck	56%	81% 128 x 72 px
car	7%	54% 128 x 72 px