Closed BurningFr closed 3 years ago
Hi, it's the best validation result. Fix the random seed, and run multiple times to report the average.
Hi, thank you for answering my question!
I find there are some missing files in ./models (alexnet.py). I run the 'train.py' on my server and the reproduction results cannot achieve as high as the paper released (PACS dataset on resnet18). If I miss something important? I have annotated some codes in class 'JigsawNewDataset' which are for jigsaw data generate process to run RSC model.
My reproduction results is 79.45% for Sketch and 80.22% for Art Painting.
I have uploaded my environment. Could you try to run the code in that environment? I attach some results I just got for Sketch and Art painting. Thanks for pointing out the legacy code in class 'JigsawNewDataset'.
sketch Dataset size: train 5446, val 616, test 3929 Step size: 24 0/42 of epoch 1/30 class : 2.189 - acc class : 7.03 [bs:128] 30/42 of epoch 1/30 class : 0.479 - acc class : 87.50 [bs:128] Accuracies on val: class : 88.96 Accuracies on test: class : 64.77 New epoch - lr: 0.008 0/42 of epoch 2/30 class : 1.116 - acc class : 58.59 [bs:128] 30/42 of epoch 2/30 class : 0.335 - acc class : 92.19 [bs:128] Accuracies on val: class : 92.05 Accuracies on test: class : 66.56 New epoch - lr: 0.008 0/42 of epoch 3/30 class : 0.646 - acc class : 72.66 [bs:128] 30/42 of epoch 3/30 class : 0.299 - acc class : 94.53 [bs:128] Accuracies on val: class : 90.75 Accuracies on test: class : 69.92 New epoch - lr: 0.008 0/42 of epoch 4/30 class : 0.513 - acc class : 71.88 [bs:128] 30/42 of epoch 4/30 class : 0.434 - acc class : 83.59 [bs:128] Accuracies on val: class : 93.51 Accuracies on test: class : 57.19 New epoch - lr: 0.008 0/42 of epoch 5/30 class : 0.324 - acc class : 89.84 [bs:128] 30/42 of epoch 5/30 class : 0.235 - acc class : 94.53 [bs:128] Accuracies on val: class : 96.59 Accuracies on test: class : 71.21 New epoch - lr: 0.008 0/42 of epoch 6/30 class : 0.156 - acc class : 98.44 [bs:128] 30/42 of epoch 6/30 class : 0.208 - acc class : 92.97 [bs:128] Accuracies on val: class : 93.83 Accuracies on test: class : 77.50 New epoch - lr: 0.008 0/42 of epoch 7/30 class : 0.243 - acc class : 92.97 [bs:128] 30/42 of epoch 7/30 class : 0.255 - acc class : 91.41 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 72.94 New epoch - lr: 0.008 0/42 of epoch 8/30 class : 0.183 - acc class : 95.31 [bs:128] 30/42 of epoch 8/30 class : 0.173 - acc class : 94.53 [bs:128] Accuracies on val: class : 94.48 Accuracies on test: class : 76.25 New epoch - lr: 0.008 0/42 of epoch 9/30 class : 0.107 - acc class : 98.44 [bs:128] 30/42 of epoch 9/30 class : 0.282 - acc class : 89.06 [bs:128] Accuracies on val: class : 94.32 Accuracies on test: class : 77.76 New epoch - lr: 0.008 0/42 of epoch 10/30 class : 0.152 - acc class : 94.53 [bs:128] 30/42 of epoch 10/30 class : 0.081 - acc class : 98.44 [bs:128] Accuracies on val: class : 95.94 Accuracies on test: class : 75.03 New epoch - lr: 0.008 0/42 of epoch 11/30 class : 0.092 - acc class : 97.66 [bs:128] 30/42 of epoch 11/30 class : 0.246 - acc class : 91.41 [bs:128] Accuracies on val: class : 95.45 Accuracies on test: class : 78.62 New epoch - lr: 0.008 0/42 of epoch 12/30 class : 0.084 - acc class : 98.44 [bs:128] 30/42 of epoch 12/30 class : 0.047 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.45 Accuracies on test: class : 72.36 New epoch - lr: 0.008 0/42 of epoch 13/30 class : 0.084 - acc class : 98.44 [bs:128] 30/42 of epoch 13/30 class : 0.146 - acc class : 96.09 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 75.44 New epoch - lr: 0.008 0/42 of epoch 14/30 class : 0.036 - acc class : 100.00 [bs:128] 30/42 of epoch 14/30 class : 0.037 - acc class : 100.00 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 73.63 New epoch - lr: 0.008 0/42 of epoch 15/30 class : 0.106 - acc class : 96.09 [bs:128] 30/42 of epoch 15/30 class : 0.083 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.48 Accuracies on test: class : 79.05 New epoch - lr: 0.008 0/42 of epoch 16/30 class : 0.053 - acc class : 99.22 [bs:128] 30/42 of epoch 16/30 class : 0.035 - acc class : 100.00 [bs:128] Accuracies on val: class : 93.99 Accuracies on test: class : 75.21 New epoch - lr: 0.008 0/42 of epoch 17/30 class : 0.064 - acc class : 98.44 [bs:128] 30/42 of epoch 17/30 class : 0.038 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.62 Accuracies on test: class : 74.62 New epoch - lr: 0.008 0/42 of epoch 18/30 class : 0.061 - acc class : 99.22 [bs:128] 30/42 of epoch 18/30 class : 0.031 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 76.10 30/42 of epoch 19/30 class : 0.057 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 77.22 New epoch - lr: 0.008 0/42 of epoch 20/30 class : 0.017 - acc class : 100.00 [bs:128] 30/42 of epoch 20/30 class : 0.023 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 76.43 New epoch - lr: 0.008 0/42 of epoch 21/30 class : 0.049 - acc class : 96.88 [bs:128] 30/42 of epoch 21/30 class : 0.077 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.81 Accuracies on test: class : 79.97 New epoch - lr: 0.008 0/42 of epoch 22/30 class : 0.013 - acc class : 100.00 [bs:128] 30/42 of epoch 22/30 class : 0.100 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.16 Accuracies on test: class : 76.69 New epoch - lr: 0.008 0/42 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128] 30/42 of epoch 23/30 class : 0.011 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 78.72 New epoch - lr: 0.008 0/42 of epoch 24/30 class : 0.068 - acc class : 98.44 [bs:128] 30/42 of epoch 24/30 class : 0.017 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.29 Accuracies on test: class : 78.44 New epoch - lr: 0.0008 0/42 of epoch 25/30 class : 0.022 - acc class : 100.00 [bs:128] 30/42 of epoch 25/30 class : 0.027 - acc class : 99.22 [bs:128] Accuracies on val: class : 96.59 Accuracies on test: class : 80.55 New epoch - lr: 0.0008 0/42 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128] 30/42 of epoch 26/30 class : 0.014 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 80.07 New epoch - lr: 0.0008 0/42 of epoch 27/30 class : 0.032 - acc class : 99.22 [bs:128] 30/42 of epoch 27/30 class : 0.059 - acc class : 98.44 [bs:128] Accuracies on val: class : 96.27 Accuracies on test: class : 80.91 New epoch - lr: 0.0008 0/42 of epoch 28/30 class : 0.023 - acc class : 100.00 [bs:128] 30/42 of epoch 28/30 class : 0.014 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 80.35 New epoch - lr: 0.0008 0/42 of epoch 29/30 class : 0.010 - acc class : 100.00 [bs:128] 30/42 of epoch 29/30 class : 0.047 - acc class : 99.22 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 81.27 New epoch - lr: 0.0008 0/42 of epoch 30/30 class : 0.059 - acc class : 100.00 [bs:128] 30/42 of epoch 30/30 class : 0.047 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.43 Accuracies on test: class : 80.33 Best val 0.967532, corresponding test 0.812675 - best test: 0.812675, best epoch: 28 It took 557.549
art_painting Dataset size: train 7137, val 806, test 2048 Step size: 24 0/55 of epoch 1/30 class : 3.264 - acc class : 0.78 [bs:128] 30/55 of epoch 1/30 class : 0.754 - acc class : 78.91 [bs:128] Accuracies on val: class : 89.21 Accuracies on test: class : 76.32 New epoch - lr: 0.008 0/55 of epoch 2/30 class : 0.825 - acc class : 66.41 [bs:128] 30/55 of epoch 2/30 class : 0.510 - acc class : 87.50 [bs:128] Accuracies on val: class : 93.42 Accuracies on test: class : 73.29 New epoch - lr: 0.008 0/55 of epoch 3/30 class : 0.423 - acc class : 85.16 [bs:128] 30/55 of epoch 3/30 class : 0.330 - acc class : 93.75 [bs:128] Accuracies on val: class : 93.18 Accuracies on test: class : 77.54 New epoch - lr: 0.008 0/55 of epoch 4/30 class : 0.277 - acc class : 93.75 [bs:128] 30/55 of epoch 4/30 class : 0.379 - acc class : 85.94 [bs:128] Accuracies on val: class : 91.56 Accuracies on test: class : 76.03 New epoch - lr: 0.008 0/55 of epoch 5/30 class : 0.296 - acc class : 88.28 [bs:128] 30/55 of epoch 5/30 class : 0.253 - acc class : 94.53 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 77.83 New epoch - lr: 0.008 0/55 of epoch 6/30 class : 0.176 - acc class : 96.88 [bs:128] 30/55 of epoch 6/30 class : 0.198 - acc class : 96.09 [bs:128] Accuracies on val: class : 94.67 Accuracies on test: class : 77.44 New epoch - lr: 0.008 0/55 of epoch 7/30 class : 0.242 - acc class : 90.62 [bs:128] 30/55 of epoch 7/30 class : 0.200 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.16 Accuracies on test: class : 79.54 New epoch - lr: 0.008 0/55 of epoch 8/30 class : 0.125 - acc class : 98.44 [bs:128] 30/55 of epoch 8/30 class : 0.153 - acc class : 96.09 [bs:128] Accuracies on val: class : 92.80 Accuracies on test: class : 75.05 New epoch - lr: 0.008 0/55 of epoch 9/30 class : 0.079 - acc class : 98.44 [bs:128] 30/55 of epoch 9/30 class : 0.089 - acc class : 99.22 [bs:128] Accuracies on val: class : 93.80 Accuracies on test: class : 75.78 New epoch - lr: 0.008 0/55 of epoch 10/30 class : 0.115 - acc class : 96.88 [bs:128] 30/55 of epoch 10/30 class : 0.131 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 78.56 New epoch - lr: 0.008 0/55 of epoch 11/30 class : 0.091 - acc class : 99.22 [bs:128] 30/55 of epoch 11/30 class : 0.113 - acc class : 96.88 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 81.25 New epoch - lr: 0.008 0/55 of epoch 12/30 class : 0.109 - acc class : 96.88 [bs:128] 30/55 of epoch 12/30 class : 0.140 - acc class : 96.88 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 74.61 New epoch - lr: 0.008 0/55 of epoch 13/30 class : 0.043 - acc class : 100.00 [bs:128] 30/55 of epoch 13/30 class : 0.038 - acc class : 100.00 [bs:128] Accuracies on val: class : 92.31 Accuracies on test: class : 80.57 New epoch - lr: 0.008 0/55 of epoch 14/30 class : 0.060 - acc class : 98.44 [bs:128] 30/55 of epoch 14/30 class : 0.123 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 76.76 New epoch - lr: 0.008 0/55 of epoch 15/30 class : 0.082 - acc class : 96.09 [bs:128] 30/55 of epoch 15/30 class : 0.046 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.53 Accuracies on test: class : 79.88 New epoch - lr: 0.008 0/55 of epoch 16/30 class : 0.040 - acc class : 98.44 [bs:128] 30/55 of epoch 16/30 class : 0.030 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.66 Accuracies on test: class : 80.57 New epoch - lr: 0.008 0/55 of epoch 17/30 class : 0.062 - acc class : 98.44 [bs:128] 30/55 of epoch 17/30 class : 0.092 - acc class : 98.44 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 78.42 New epoch - lr: 0.008 0/55 of epoch 18/30 class : 0.029 - acc class : 100.00 [bs:128] 30/55 of epoch 18/30 class : 0.042 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.04 Accuracies on test: class : 73.34 New epoch - lr: 0.008 0/55 of epoch 19/30 class : 0.079 - acc class : 96.88 [bs:128] 30/55 of epoch 19/30 class : 0.116 - acc class : 95.31 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 76.81 New epoch - lr: 0.008 0/55 of epoch 20/30 class : 0.034 - acc class : 99.22 [bs:128] 30/55 of epoch 20/30 class : 0.019 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.15 Accuracies on test: class : 79.10 New epoch - lr: 0.008 0/55 of epoch 21/30 class : 0.032 - acc class : 99.22 [bs:128] 30/55 of epoch 21/30 class : 0.022 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 76.32 New epoch - lr: 0.008 0/55 of epoch 22/30 class : 0.042 - acc class : 98.44 [bs:128] 30/55 of epoch 22/30 class : 0.024 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 81.64 New epoch - lr: 0.008 0/55 of epoch 23/30 class : 0.034 - acc class : 99.22 [bs:128] 30/55 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 81.84 New epoch - lr: 0.008 0/55 of epoch 24/30 class : 0.009 - acc class : 100.00 [bs:128] 30/55 of epoch 24/30 class : 0.032 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 78.08 New epoch - lr: 0.0008 0/55 of epoch 25/30 class : 0.019 - acc class : 99.22 [bs:128] 30/55 of epoch 25/30 class : 0.010 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.28 Accuracies on test: class : 81.88 New epoch - lr: 0.0008 0/55 of epoch 26/30 class : 0.025 - acc class : 100.00 [bs:128] 30/55 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.53 Accuracies on test: class : 81.88 New epoch - lr: 0.0008 0/55 of epoch 27/30 class : 0.013 - acc class : 100.00 [bs:128] 30/55 of epoch 27/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.40 Accuracies on test: class : 82.37 New epoch - lr: 0.0008 0/55 of epoch 28/30 class : 0.028 - acc class : 100.00 [bs:128] 30/55 of epoch 28/30 class : 0.005 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.40 Accuracies on test: class : 81.10 New epoch - lr: 0.0008 0/55 of epoch 29/30 class : 0.035 - acc class : 100.00 [bs:128] 30/55 of epoch 29/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 97.02 Accuracies on test: class : 82.81 New epoch - lr: 0.0008 0/55 of epoch 30/30 class : 0.014 - acc class : 100.00 [bs:128] 30/55 of epoch 30/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.65 Accuracies on test: class : 83.20 Best val 0.970223, corresponding test 0.828125 - best test: 0.832031, best epoch: 28 It took 562.823
Thank you for sharing your results. I fix the environment the same with your 'environment.yml' and run the PACS dataset with resnet18 for 5 times.
best val PACS | a | c | s | p | avg |
---|---|---|---|---|---|
1 | 78.7109 | 77.4744 | 79.6386 | 94.4311 | 82.56375 |
2 | 80.4199 | 79.3089 | 80.4276 | 94.7305 | 83.72173 |
3 | 79.8828 | 78.8823 | 77.8061 | 93.9521 | 82.63083 |
4 | 77.9297 | 78.2423 | 74.7773 | 94.6108 | 81.39003 |
5 | 81.2012 | 75 | 75.6936 | 94.8503 | 81.68628 |
avg | 79.6289 | 77.78158 | 77.66864 | 94.51496 | 82.39852 |
best test PACS | a | c | s | p | avg |
---|---|---|---|---|---|
1 | 81.5918 | 78.4983 | 80.1985 | 94.6707 | 83.73983 |
2 | 82.0801 | 80.5887 | 80.6821 | 95.1497 | 84.62515 |
3 | 81.6406 | 79.8208 | 78.3151 | 95.2695 | 83.7615 |
4 | 82.1777 | 81.2713 | 81.0384 | 94.9701 | 84.86438 |
5 | 81.2012 | 79.029 | 82.0565 | 95.0299 | 84.32915 |
avg | 81.73828 | 79.84162 | 80.45812 | 95.01798 | 84.264 |
I find the results is not stable with random initialization.
Hi, I run one experiment at a time in the server, and the best Val result usually appears at the last 5 epoch. It should be more stable and better. I will try to fix the randomness anyway.
Got it! Thank you.
In train.py(L145), I find the code also records the best performance of test domain in logger.save_best().
Could you point out which result the paper uses, is it the best performance of test domain (test_res.max()) or the best validation performance (test_res[idx_best])?
Thanks a lot!