Low Recall rate

[fjscoelho]

Description

I'm getting low TPR's with the NetHALOC. I trained the network with the original Dataset (Mediterranean) and made cross validation. Here are the results:

Results

1. OriginalModel_40EP_10ES_trainDS2_valDS1: [FULLSTATS COMPUTED 0.9965846994535519 0.5384615384615384 0.0013722126929674098 173.96081376075745]
2. OriginalModel_40EP_10ES_trainDS1_valDS2: [FULLSTATS COMPUTED 0.9960724043715847 0.4230769230769231 0.0013722126929674098 206.42046976089478]
3. OriginalModel_40EP_10ES_trainDS3_valDS2: [FULLSTATS COMPUTED 0.9931693989071039 0.4117647058823529 0.002294630564479119 150.2515733242035]
4. OriginalModel_40EP_10ES_trainDS2_valDS3: [FULLSTATS COMPUTED 0.9927140255009107 0.4411764705882353 0.0029830197338228544 147.43460416793823]
5. OriginalModel_40EP_10ES_trainDS1_valDS3: [FULLSTATS COMPUTED 0.9938983050847457 0.19230769230769232 0.00136394635144351 183.1872260570526]

Obs.: The models was trained with max of 40 epochs and an early stop configuration to 10 epochs without loss_validation evolution.

Additional Information

• OS: Ubuntu 22.04.4 LTS
• Python 3.10.12
• Tensorflow 2.17.0
• Keras (Inside Tensorflow) 3.5.0
• OpenCV2 4.10.0
• SciKit-Image 0.24.0
• SciKit-Learn 1.5.1

[fjscoelho]

Possible solutions

Verify the effect of change the imageMatcher Threshold: the value setted by the authors is 0.75. Model used and initial metrics: OriginalModel_40EP_10ES_trainDS2_valDS1: [FULLSTATS COMPUTED 0.9965846994535519 0.5384615384615384 0.0013722126929674098 173.96081376075745]

[fjscoelho]

Possible solutions

Verify the effect of change the imageMatcher Threshold: the value setted by the authors is 0.75. Model used and initial metrics: OriginalModel_40EP_10ES_trainDS2_valDS1: [FULLSTATS COMPUTED 0.9965846994535519 0.5384615384615384 0.0013722126929674098 173.96081376075745]

matchThreshold=.80 -> [FULLSTATS COMPUTED 0.9952185792349727 0.5 0.002572898799313894 171.44175267219543]: All metrics got worse.

matchThreshold=.85 -> [FULLSTATS COMPUTED 0.9965846994535519 0.46153846153846156 0.0010291595197255575 204.3911862373352]: ACC and Fallout are better, but the TPR (Recall) is worse.

matchThreshold=.90 -> [FULLSTATS COMPUTED 0.9955601092896175 0.23076923076923078 0.0010291595197255575 209.43870973587036] : The TPR is even worse.

matchThreshold=.95 -> [FULLSTATS COMPUTED 0.9950478142076503 0.038461538461538464 0.0006861063464837049 210.66262125968933]: Bad results.

matchThreshold=.70 -> [FULLSTATS COMPUTED 0.9964139344262295 0.4230769230769231 0.0010291595197255575 200.06368327140808]: TPR is still worse than the initial one.

matchThreshold=.65 -> [FULLSTATS COMPUTED 0.9960724043715847 0.34615384615384615 0.0010291595197255575 162.35925364494324]

matchThreshold=.60 -> [FULLSTATS COMPUTED 0.9959016393442623 0.3076923076923077 0.0010291595197255575 165.54050087928772]:

matchThreshold=.50 -> [FULLSTATS COMPUTED 0.9955601092896175 0.23076923076923078 0.0010291595197255575 168.825124502182]

Conclusion

Increasing or decreasing the Imagematcher Threshold dind't improve the recall.

[fjscoelho]

Possible Solutions

Increase the epochs number. I have trained the NN with 100 ephocs, without EarlyStoop. In this training I have used the dataset2 for training, dataset1 for validation and the dataset3 for test, because this configuration presented the better accuracy result in the previous test.

[FULLSTATS COMPUTED 0.9959016393442623 0.46153846153846156 0.0017152658662092624 109.646812915802]

There was no improvement in accuracy and observing the training evolution we can note that the loss_val stagnated after the initial epochs.