fails to reproduce

[seekingup]

Hi,

Thanks for the nice work and open source code.

I cloned the code yesterday and ran the consistency setting in CIFAR-10 with the command provided in the README.md:

python train.py --dataset cifar10 --num-max 500 --num-max-u 4000 --arch wideresnet --batch-size 64 --lr 0.03 --seed 0 --imb-ratio-label 100 --imb-ratio-unlabel 100 --ema-u 0.99 --out out/cifar-10/N500_M4000/consistent

However, I failed to achieve similar performance as reported in the paper.

Here is my output:

current epoch: 500
05/19/2023 06:10:55 - INFO - main - top-1 acc: 74.44
05/19/2023 06:10:55 - INFO - main - top-5 acc: 97.74
05/19/2023 06:10:55 - INFO - main - Best top-1 acc: 76.66
05/19/2023 06:10:55 - INFO - main - Mean top-1 acc: 74.89

The reported performance in this setting of the paper is 81.6%. Is there something I missed?

I used a single GPU with PyTorch 1.4.

[Gank0078]

Thank you for your interest in our work. In our method ACR, there are standard branch and balanced branch. The results we report in the paper are all from balanced branch, while the results printed during training are the results of the standard branch. So you can refer to the complete results of the two branches recorded in the log.txt file during training. Of course, we will also modify the relevant parts in the code to print the results of the balanced branch.

[seekingup]

Thanks for your quick reply. After checking the log.txt, the best balance branch accuracy is 81.5, which is close to 81.6%.

Top1 acc    Top5 acc    Best Top1 acc   Top1_b acc  Top5_b acc  Best Top1_b acc 
75.330000   98.630000   75.330000   81.500000   99.010000   81.500000 (ep263)
74.440000   97.740000   76.660000   74.430000   97.990000   81.500000 (ep500)

[weilegexiang]

Why the best accuracy is reported other than the last accuracy?

[Gank0078]

Why the best accuracy is reported other than the last accuracy?

Some previous LTSSL methods record the results of the optimal epoch, such as ABC. And we have to admit that our model will converge fast in the consistent setting, but considering that the previous methods have been trained for 500 epochs, we still follow this point. In my opinion, our model will accumulate more bias in the later stage of training, especially for consistent setting, which will lead to a certain degree of performance decline. This may be the direction of our improvement in the future.

[weilegexiang]

Why the best accuracy is reported other than the last accuracy?

Some previous LTSSL methods record the results of the optimal epoch, such as ABC. And we have to admit that our model will converge fast in the consistent setting, but considering that the previous methods have been trained for 500 epochs, we still follow this point. In my opinion, our model will accumulate more bias in the later stage of training, especially for consistent setting, which will lead to a certain degree of performance decline. This may be the direction of our improvement in the future.

Thank you for your kind reply, I will learn from your method in my future work, and thank you for source code sharing again.