search

eccv22-ood-workshop / ROBIN-dataset

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts
http://www.ood-cv.org/
46 stars 6 forks source link

Baseline code problem [Reproduce Acc] #35

Open khawar-islam opened 1 year ago

khawar-islam commented 1 year ago

Dear @DTennant,

I have used your baseline to reproduce the results mentioned in the paper but I am unable to reproduce even a single result and after 1st epoch, I got 100% accuracy. What is the problem? please let us know

Regards, Khawar Islam

=> creating model 'resnet50'
Epoch: [0][  0/136] Time  2.752 ( 2.752)    Data  0.257 ( 0.257)    Loss 7.8677e+00 (7.8677e+00)    Acc@1   0.00 (  0.00)   Acc@5   0.00 (  0.00)
Epoch: [0][ 10/136] Time  0.172 ( 0.397)    Data  0.000 ( 0.024)    Loss 1.5089e+00 (5.2092e+00)    Acc@1 100.00 ( 45.60)   Acc@5 100.00 ( 49.43)
Epoch: [0][ 20/136] Time  0.173 ( 0.291)    Data  0.000 ( 0.016)    Loss 8.8183e-03 (2.8527e+00)    Acc@1 100.00 ( 71.50)   Acc@5 100.00 ( 73.51)
Epoch: [0][ 30/136] Time  0.172 ( 0.253)    Data  0.000 ( 0.014)    Loss 1.4902e-03 (1.9330e+00)    Acc@1 100.00 ( 80.70)   Acc@5 100.00 ( 82.06)
Epoch: [0][ 40/136] Time  0.173 ( 0.233)    Data  0.000 ( 0.013)    Loss 1.6841e-04 (1.4616e+00)    Acc@1 100.00 ( 85.40)   Acc@5 100.00 ( 86.43)
Epoch: [0][ 50/136] Time  0.172 ( 0.221)    Data  0.000 ( 0.012)    Loss 2.0796e-04 (1.1751e+00)    Acc@1 100.00 ( 88.27)   Acc@5 100.00 ( 89.09)
Epoch: [0][ 60/136] Time  0.173 ( 0.213)    Data  0.000 ( 0.011)    Loss 2.7306e-04 (9.8248e-01)    Acc@1 100.00 ( 90.19)   Acc@5 100.00 ( 90.88)
Epoch: [0][ 70/136] Time  0.174 ( 0.208)    Data  0.000 ( 0.011)    Loss 2.8168e-04 (8.4412e-01)    Acc@1 100.00 ( 91.57)   Acc@5 100.00 ( 92.17)
Epoch: [0][ 80/136] Time  0.175 ( 0.204)    Data  0.000 ( 0.010)    Loss 1.2127e-04 (7.3993e-01)    Acc@1 100.00 ( 92.61)   Acc@5 100.00 ( 93.13)
Epoch: [0][ 90/136] Time  0.173 ( 0.200)    Data  0.000 ( 0.010)    Loss 1.4042e-04 (6.5863e-01)    Acc@1 100.00 ( 93.42)   Acc@5 100.00 ( 93.89)
Epoch: [0][100/136] Time  0.175 ( 0.198)    Data  0.000 ( 0.010)    Loss 1.2365e-04 (5.9343e-01)    Acc@1 100.00 ( 94.07)   Acc@5 100.00 ( 94.49)
Epoch: [0][110/136] Time  0.174 ( 0.196)    Data  0.000 ( 0.010)    Loss 1.5705e-04 (5.3998e-01)    Acc@1 100.00 ( 94.61)   Acc@5 100.00 ( 94.99)
Epoch: [0][120/136] Time  0.159 ( 0.193)    Data  0.000 ( 0.010)    Loss 1.6686e-04 (4.9537e-01)    Acc@1 100.00 ( 95.05)   Acc@5 100.00 ( 95.40)
Epoch: [0][130/136] Time  0.161 ( 0.191)    Data  0.000 ( 0.010)    Loss 1.4597e-04 (4.5757e-01)    Acc@1 100.00 ( 95.43)   Acc@5 100.00 ( 95.75)
Test shape: [ 0/16] Time  0.273 ( 0.273)    Loss 1.9905e-04 (1.9905e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
Test shape: [10/16] Time  0.051 ( 0.072)    Loss 1.5740e-04 (1.1862e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
 *   Acc@1 100.000 Acc@5 100.000
Test pose: [ 0/17]  Time  0.309 ( 0.309)    Loss 1.3270e-04 (1.3270e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
Test pose: [10/17]  Time  0.053 ( 0.075)    Loss 1.2837e-04 (1.2978e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
 *   Acc@1 100.000 Acc@5 100.000
Test texture: [0/9] Time  0.489 ( 0.489)    Loss 2.7557e-05 (2.7557e-05)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
 *   Acc@1 100.000 Acc@5 100.000
Test context: [0/9] Time  0.522 ( 0.522)    Loss 2.5978e-05 (2.5978e-05)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
 *   Acc@1 100.000 Acc@5 100.000
Test weather: [ 0/22]   Time  0.314 ( 0.314)    Loss 4.1471e-04 (4.1471e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
Test weather: [10/22]   Time  0.053 ( 0.090)    Loss 1.8216e-04 (2.5321e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
Test weather: [20/22]   Time  0.050 ( 0.072)    Loss 8.3286e-05 (2.1466e-04)    Acc@1 100.00 (100.00)   Acc@5 100.00 (100.00)
 *   Acc@1 100.000 Acc@5 100.000