The result record in train.py has two items.

[BurningFr]

In train.py(L145), I find the code also records the best performance of test domain in logger.save_best().

Could you point out which result the paper uses, is it the best performance of test domain (test_res.max()) or the best validation performance (test_res[idx_best])?

Thanks a lot!

[Justinhzy]

Hi, it's the best validation result. Fix the random seed, and run multiple times to report the average.

[BurningFr]

Hi, thank you for answering my question!

I find there are some missing files in ./models (alexnet.py). I run the 'train.py' on my server and the reproduction results cannot achieve as high as the paper released (PACS dataset on resnet18). If I miss something important? I have annotated some codes in class 'JigsawNewDataset' which are for jigsaw data generate process to run RSC model.

My reproduction results is 79.45% for Sketch and 80.22% for Art Painting.

[Justinhzy]

I have uploaded my environment. Could you try to run the code in that environment? I attach some results I just got for Sketch and Art painting. Thanks for pointing out the legacy code in class 'JigsawNewDataset'.

sketch Dataset size: train 5446, val 616, test 3929 Step size: 24 0/42 of epoch 1/30 class : 2.189 - acc class : 7.03 [bs:128] 30/42 of epoch 1/30 class : 0.479 - acc class : 87.50 [bs:128] Accuracies on val: class : 88.96 Accuracies on test: class : 64.77 New epoch - lr: 0.008 0/42 of epoch 2/30 class : 1.116 - acc class : 58.59 [bs:128] 30/42 of epoch 2/30 class : 0.335 - acc class : 92.19 [bs:128] Accuracies on val: class : 92.05 Accuracies on test: class : 66.56 New epoch - lr: 0.008 0/42 of epoch 3/30 class : 0.646 - acc class : 72.66 [bs:128] 30/42 of epoch 3/30 class : 0.299 - acc class : 94.53 [bs:128] Accuracies on val: class : 90.75 Accuracies on test: class : 69.92 New epoch - lr: 0.008 0/42 of epoch 4/30 class : 0.513 - acc class : 71.88 [bs:128] 30/42 of epoch 4/30 class : 0.434 - acc class : 83.59 [bs:128] Accuracies on val: class : 93.51 Accuracies on test: class : 57.19 New epoch - lr: 0.008 0/42 of epoch 5/30 class : 0.324 - acc class : 89.84 [bs:128] 30/42 of epoch 5/30 class : 0.235 - acc class : 94.53 [bs:128] Accuracies on val: class : 96.59 Accuracies on test: class : 71.21 New epoch - lr: 0.008 0/42 of epoch 6/30 class : 0.156 - acc class : 98.44 [bs:128] 30/42 of epoch 6/30 class : 0.208 - acc class : 92.97 [bs:128] Accuracies on val: class : 93.83 Accuracies on test: class : 77.50 New epoch - lr: 0.008 0/42 of epoch 7/30 class : 0.243 - acc class : 92.97 [bs:128] 30/42 of epoch 7/30 class : 0.255 - acc class : 91.41 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 72.94 New epoch - lr: 0.008 0/42 of epoch 8/30 class : 0.183 - acc class : 95.31 [bs:128] 30/42 of epoch 8/30 class : 0.173 - acc class : 94.53 [bs:128] Accuracies on val: class : 94.48 Accuracies on test: class : 76.25 New epoch - lr: 0.008 0/42 of epoch 9/30 class : 0.107 - acc class : 98.44 [bs:128] 30/42 of epoch 9/30 class : 0.282 - acc class : 89.06 [bs:128] Accuracies on val: class : 94.32 Accuracies on test: class : 77.76 New epoch - lr: 0.008 0/42 of epoch 10/30 class : 0.152 - acc class : 94.53 [bs:128] 30/42 of epoch 10/30 class : 0.081 - acc class : 98.44 [bs:128] Accuracies on val: class : 95.94 Accuracies on test: class : 75.03 New epoch - lr: 0.008 0/42 of epoch 11/30 class : 0.092 - acc class : 97.66 [bs:128] 30/42 of epoch 11/30 class : 0.246 - acc class : 91.41 [bs:128] Accuracies on val: class : 95.45 Accuracies on test: class : 78.62 New epoch - lr: 0.008 0/42 of epoch 12/30 class : 0.084 - acc class : 98.44 [bs:128] 30/42 of epoch 12/30 class : 0.047 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.45 Accuracies on test: class : 72.36 New epoch - lr: 0.008 0/42 of epoch 13/30 class : 0.084 - acc class : 98.44 [bs:128] 30/42 of epoch 13/30 class : 0.146 - acc class : 96.09 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 75.44 New epoch - lr: 0.008 0/42 of epoch 14/30 class : 0.036 - acc class : 100.00 [bs:128] 30/42 of epoch 14/30 class : 0.037 - acc class : 100.00 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 73.63 New epoch - lr: 0.008 0/42 of epoch 15/30 class : 0.106 - acc class : 96.09 [bs:128] 30/42 of epoch 15/30 class : 0.083 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.48 Accuracies on test: class : 79.05 New epoch - lr: 0.008 0/42 of epoch 16/30 class : 0.053 - acc class : 99.22 [bs:128] 30/42 of epoch 16/30 class : 0.035 - acc class : 100.00 [bs:128] Accuracies on val: class : 93.99 Accuracies on test: class : 75.21 New epoch - lr: 0.008 0/42 of epoch 17/30 class : 0.064 - acc class : 98.44 [bs:128] 30/42 of epoch 17/30 class : 0.038 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.62 Accuracies on test: class : 74.62 New epoch - lr: 0.008 0/42 of epoch 18/30 class : 0.061 - acc class : 99.22 [bs:128] 30/42 of epoch 18/30 class : 0.031 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 76.10 30/42 of epoch 19/30 class : 0.057 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.97 Accuracies on test: class : 77.22 New epoch - lr: 0.008 0/42 of epoch 20/30 class : 0.017 - acc class : 100.00 [bs:128] 30/42 of epoch 20/30 class : 0.023 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 76.43 New epoch - lr: 0.008 0/42 of epoch 21/30 class : 0.049 - acc class : 96.88 [bs:128] 30/42 of epoch 21/30 class : 0.077 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.81 Accuracies on test: class : 79.97 New epoch - lr: 0.008 0/42 of epoch 22/30 class : 0.013 - acc class : 100.00 [bs:128] 30/42 of epoch 22/30 class : 0.100 - acc class : 98.44 [bs:128] Accuracies on val: class : 94.16 Accuracies on test: class : 76.69 New epoch - lr: 0.008 0/42 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128] 30/42 of epoch 23/30 class : 0.011 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.13 Accuracies on test: class : 78.72 New epoch - lr: 0.008 0/42 of epoch 24/30 class : 0.068 - acc class : 98.44 [bs:128] 30/42 of epoch 24/30 class : 0.017 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.29 Accuracies on test: class : 78.44 New epoch - lr: 0.0008 0/42 of epoch 25/30 class : 0.022 - acc class : 100.00 [bs:128] 30/42 of epoch 25/30 class : 0.027 - acc class : 99.22 [bs:128] Accuracies on val: class : 96.59 Accuracies on test: class : 80.55 New epoch - lr: 0.0008 0/42 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128] 30/42 of epoch 26/30 class : 0.014 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 80.07 New epoch - lr: 0.0008 0/42 of epoch 27/30 class : 0.032 - acc class : 99.22 [bs:128] 30/42 of epoch 27/30 class : 0.059 - acc class : 98.44 [bs:128] Accuracies on val: class : 96.27 Accuracies on test: class : 80.91 New epoch - lr: 0.0008 0/42 of epoch 28/30 class : 0.023 - acc class : 100.00 [bs:128] 30/42 of epoch 28/30 class : 0.014 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 80.35 New epoch - lr: 0.0008 0/42 of epoch 29/30 class : 0.010 - acc class : 100.00 [bs:128] 30/42 of epoch 29/30 class : 0.047 - acc class : 99.22 [bs:128] Accuracies on val: class : 96.75 Accuracies on test: class : 81.27 New epoch - lr: 0.0008 0/42 of epoch 30/30 class : 0.059 - acc class : 100.00 [bs:128] 30/42 of epoch 30/30 class : 0.047 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.43 Accuracies on test: class : 80.33 Best val 0.967532, corresponding test 0.812675 - best test: 0.812675, best epoch: 28 It took 557.549

art_painting Dataset size: train 7137, val 806, test 2048 Step size: 24 0/55 of epoch 1/30 class : 3.264 - acc class : 0.78 [bs:128] 30/55 of epoch 1/30 class : 0.754 - acc class : 78.91 [bs:128] Accuracies on val: class : 89.21 Accuracies on test: class : 76.32 New epoch - lr: 0.008 0/55 of epoch 2/30 class : 0.825 - acc class : 66.41 [bs:128] 30/55 of epoch 2/30 class : 0.510 - acc class : 87.50 [bs:128] Accuracies on val: class : 93.42 Accuracies on test: class : 73.29 New epoch - lr: 0.008 0/55 of epoch 3/30 class : 0.423 - acc class : 85.16 [bs:128] 30/55 of epoch 3/30 class : 0.330 - acc class : 93.75 [bs:128] Accuracies on val: class : 93.18 Accuracies on test: class : 77.54 New epoch - lr: 0.008 0/55 of epoch 4/30 class : 0.277 - acc class : 93.75 [bs:128] 30/55 of epoch 4/30 class : 0.379 - acc class : 85.94 [bs:128] Accuracies on val: class : 91.56 Accuracies on test: class : 76.03 New epoch - lr: 0.008 0/55 of epoch 5/30 class : 0.296 - acc class : 88.28 [bs:128] 30/55 of epoch 5/30 class : 0.253 - acc class : 94.53 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 77.83 New epoch - lr: 0.008 0/55 of epoch 6/30 class : 0.176 - acc class : 96.88 [bs:128] 30/55 of epoch 6/30 class : 0.198 - acc class : 96.09 [bs:128] Accuracies on val: class : 94.67 Accuracies on test: class : 77.44 New epoch - lr: 0.008 0/55 of epoch 7/30 class : 0.242 - acc class : 90.62 [bs:128] 30/55 of epoch 7/30 class : 0.200 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.16 Accuracies on test: class : 79.54 New epoch - lr: 0.008 0/55 of epoch 8/30 class : 0.125 - acc class : 98.44 [bs:128] 30/55 of epoch 8/30 class : 0.153 - acc class : 96.09 [bs:128] Accuracies on val: class : 92.80 Accuracies on test: class : 75.05 New epoch - lr: 0.008 0/55 of epoch 9/30 class : 0.079 - acc class : 98.44 [bs:128] 30/55 of epoch 9/30 class : 0.089 - acc class : 99.22 [bs:128] Accuracies on val: class : 93.80 Accuracies on test: class : 75.78 New epoch - lr: 0.008 0/55 of epoch 10/30 class : 0.115 - acc class : 96.88 [bs:128] 30/55 of epoch 10/30 class : 0.131 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 78.56 New epoch - lr: 0.008 0/55 of epoch 11/30 class : 0.091 - acc class : 99.22 [bs:128] 30/55 of epoch 11/30 class : 0.113 - acc class : 96.88 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 81.25 New epoch - lr: 0.008 0/55 of epoch 12/30 class : 0.109 - acc class : 96.88 [bs:128] 30/55 of epoch 12/30 class : 0.140 - acc class : 96.88 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 74.61 New epoch - lr: 0.008 0/55 of epoch 13/30 class : 0.043 - acc class : 100.00 [bs:128] 30/55 of epoch 13/30 class : 0.038 - acc class : 100.00 [bs:128] Accuracies on val: class : 92.31 Accuracies on test: class : 80.57 New epoch - lr: 0.008 0/55 of epoch 14/30 class : 0.060 - acc class : 98.44 [bs:128] 30/55 of epoch 14/30 class : 0.123 - acc class : 95.31 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 76.76 New epoch - lr: 0.008 0/55 of epoch 15/30 class : 0.082 - acc class : 96.09 [bs:128] 30/55 of epoch 15/30 class : 0.046 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.53 Accuracies on test: class : 79.88 New epoch - lr: 0.008 0/55 of epoch 16/30 class : 0.040 - acc class : 98.44 [bs:128] 30/55 of epoch 16/30 class : 0.030 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.66 Accuracies on test: class : 80.57 New epoch - lr: 0.008 0/55 of epoch 17/30 class : 0.062 - acc class : 98.44 [bs:128] 30/55 of epoch 17/30 class : 0.092 - acc class : 98.44 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 78.42 New epoch - lr: 0.008 0/55 of epoch 18/30 class : 0.029 - acc class : 100.00 [bs:128] 30/55 of epoch 18/30 class : 0.042 - acc class : 99.22 [bs:128] Accuracies on val: class : 94.04 Accuracies on test: class : 73.34 New epoch - lr: 0.008 0/55 of epoch 19/30 class : 0.079 - acc class : 96.88 [bs:128] 30/55 of epoch 19/30 class : 0.116 - acc class : 95.31 [bs:128] Accuracies on val: class : 94.54 Accuracies on test: class : 76.81 New epoch - lr: 0.008 0/55 of epoch 20/30 class : 0.034 - acc class : 99.22 [bs:128] 30/55 of epoch 20/30 class : 0.019 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.15 Accuracies on test: class : 79.10 New epoch - lr: 0.008 0/55 of epoch 21/30 class : 0.032 - acc class : 99.22 [bs:128] 30/55 of epoch 21/30 class : 0.022 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 76.32 New epoch - lr: 0.008 0/55 of epoch 22/30 class : 0.042 - acc class : 98.44 [bs:128] 30/55 of epoch 22/30 class : 0.024 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.41 Accuracies on test: class : 81.64 New epoch - lr: 0.008 0/55 of epoch 23/30 class : 0.034 - acc class : 99.22 [bs:128] 30/55 of epoch 23/30 class : 0.013 - acc class : 100.00 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 81.84 New epoch - lr: 0.008 0/55 of epoch 24/30 class : 0.009 - acc class : 100.00 [bs:128] 30/55 of epoch 24/30 class : 0.032 - acc class : 99.22 [bs:128] Accuracies on val: class : 95.78 Accuracies on test: class : 78.08 New epoch - lr: 0.0008 0/55 of epoch 25/30 class : 0.019 - acc class : 99.22 [bs:128] 30/55 of epoch 25/30 class : 0.010 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.28 Accuracies on test: class : 81.88 New epoch - lr: 0.0008 0/55 of epoch 26/30 class : 0.025 - acc class : 100.00 [bs:128] 30/55 of epoch 26/30 class : 0.012 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.53 Accuracies on test: class : 81.88 New epoch - lr: 0.0008 0/55 of epoch 27/30 class : 0.013 - acc class : 100.00 [bs:128] 30/55 of epoch 27/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.40 Accuracies on test: class : 82.37 New epoch - lr: 0.0008 0/55 of epoch 28/30 class : 0.028 - acc class : 100.00 [bs:128] 30/55 of epoch 28/30 class : 0.005 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.40 Accuracies on test: class : 81.10 New epoch - lr: 0.0008 0/55 of epoch 29/30 class : 0.035 - acc class : 100.00 [bs:128] 30/55 of epoch 29/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 97.02 Accuracies on test: class : 82.81 New epoch - lr: 0.0008 0/55 of epoch 30/30 class : 0.014 - acc class : 100.00 [bs:128] 30/55 of epoch 30/30 class : 0.009 - acc class : 100.00 [bs:128] Accuracies on val: class : 96.65 Accuracies on test: class : 83.20 Best val 0.970223, corresponding test 0.828125 - best test: 0.832031, best epoch: 28 It took 562.823

[BurningFr]

Thank you for sharing your results. I fix the environment the same with your 'environment.yml' and run the PACS dataset with resnet18 for 5 times.

best val PACS	a	c	s	p	avg
1	78.7109	77.4744	79.6386	94.4311	82.56375
2	80.4199	79.3089	80.4276	94.7305	83.72173
3	79.8828	78.8823	77.8061	93.9521	82.63083
4	77.9297	78.2423	74.7773	94.6108	81.39003
5	81.2012	75	75.6936	94.8503	81.68628
avg	79.6289	77.78158	77.66864	94.51496	82.39852

best test PACS	a	c	s	p	avg
1	81.5918	78.4983	80.1985	94.6707	83.73983
2	82.0801	80.5887	80.6821	95.1497	84.62515
3	81.6406	79.8208	78.3151	95.2695	83.7615
4	82.1777	81.2713	81.0384	94.9701	84.86438
5	81.2012	79.029	82.0565	95.0299	84.32915
avg	81.73828	79.84162	80.45812	95.01798	84.264

I find the results is not stable with random initialization.

[Justinhzy]

Hi, I run one experiment at a time in the server, and the best Val result usually appears at the last 5 epoch. It should be more stable and better. I will try to fix the randomness anyway.

[BurningFr]

Got it! Thank you.