search

icarofua / vehicle-rear

Vehicle-Rear: A New Dataset to Explore Feature Fusion For Vehicle Identification Using Convolutional Neural Networks
Apache License 2.0
111 stars 19 forks source link

Bug in 5-fold cross-validation #20

Closed adich23 closed 4 years ago

adich23 commented 4 years ago

Hi, The code used for k-fold cross validation in all scripts seems buggy. Only the Validation set is correctly set in the loops, remaining Train and Test Sets have repeated data in 3 loops. Below is the Set-data splits according to the code vs what is written in the paper.

Code Loop:0 Valid: Set01 Train: Set02 Set03 Test: Set04 Set05

Loop:1 Valid: Set02 Train: Set01 Set03 Test: Set04 Set05

Loop:2 Valid: Set03 Train: Set01 Set02 Test: Set04 Set05

Loop:3 Valid: Set04 Train: Set01 Set02 Test: Set03 Set05

Loop:4 Valid: Set05 Train: Set01 Set02 Test: Set03 Set04

Expected Loop:0 Valid: Set03 Train: Set01 Set02 Test: Set04 Set05

Loop:1 Valid: Set04 Train: Set02 Set03 Test: Set05 Set01

Loop:2 Valid: Set05 Train: Set03 Set04 Test: Set01 Set02

Loop:3 Valid: Set01 Train: Set04 Set05 Test: Set02 Set03

Loop:4 Valid: Set02 Train: Set05 Set01 Test: Set03 Set04

adich23 commented 4 years ago

I am getting very low F-1 and Recall scores with corrected Cross-validation loop vs the given code and what is claimed in the paper for the Two Stream (Shape + Plate). Can you please check?

adich23 commented 4 years ago

@icarofua should I correct the code in one of the script and submit for review? And if it is correct I will make similar modification in all of them.

icarofua commented 4 years ago

@adich23 , Thanks for the clarification of the bug and help. I am doing the changes in the scripts and testing with the new configuration, when everything is correct I will push in the git.

adich23 commented 4 years ago

Just FYI these are the performance score for siamese_two_stream.py - Shape + Plate architecture, that I have tried -

icarofua commented 4 years ago

@adich23, thanks for share.

adich23 commented 4 years ago

Hi @icarofua did you get a chance to test this?

icarofua commented 4 years ago

Hi @adich23, yes. I created a function to provide the right KFold, now I am training the new models with this new function, testing and collecting the new results. Apparently some results are different as you reported. I will update this in the github soon.

icarofua commented 4 years ago

Hi @adich23 , the bug is fixed and the code is updated.