baseline for long-tail cifar10 is 77.47%

[hbsz123]

Because my implementation for cb-focal in pytorch can only reach 77% accuracy for long-tail cifar10. I have ran your source code without any change. But the baseline is 77.5% (My baseline for pytorch in actually 75%.), which is almost 2.7% higher than the results reported in your paper. Does this mean cb-focal is only 2% higher than baseline?

[richardaecn]

Hi @hbsz123, from the screenshot, it looks like you are using softmax, not class-balanced focal loss. For the performance on CIFAR-10, please refer to Figure 8 of Appendix A in the paper (https://arxiv.org/abs/1901.05555).

[hbsz123]

Hi @hbsz123, from the screenshot, it looks like you are using softmax, not class-balanced focal loss. For the performance on CIFAR-10, please refer to Figure 8 of Appendix A in the paper (https://arxiv.org/abs/1901.05555).

Thanks for your kind reply. I use softmax to train actually.I found that the baseline trained with softmax in your paper was 74.81%, while I retrained the baseline and got 77.45%. Does this mean the baseline result reported in your paper is too low?

[richardaecn]

I re-ran the code on CIFAR-10 (imbalance factor: 50) and achieved 75.98% for vanilla softmax and 79.18% for class-balanced softmax with beta=0.9999 with the last checkpoint.

Compared with the reported numbers of 74.81% and 78.05%, the performances for both models seem improved. It could be due to the randomness of training the model or the different version of TF, CUDA and cuDNN. I ran the code with TF 1.14, CUDA 10.0 and cnDNN 7.6.1.

However, the improvement of class-balanced loss is consistent - reported in the paper: 78.05% - 74.81% = 3.24%; new ran: 79.18% - 75.98% = 3.2%.

[hbsz123]

I re-ran the code on CIFAR-10 (imbalance factor: 50) and achieved 75.98% for vanilla softmax and 79.18% for class-balanced softmax with beta=0.9999 with the last checkpoint.

Compared with the reported numbers of 74.81% and 78.05%, the performances for both models seem improved. It could be due to the randomness of training the model or the different version of TF, CUDA and cuDNN. I ran the code with TF 1.14, CUDA 10.0 and cnDNN 7.6.1.

However, the improvement of class-balanced loss is consistent - reported in the paper: 78.05% - 74.81% = 3.24%; new ran: 79.18% - 75.98% = 3.2%.

Could you please also run the class balanced focal loss? I also ran the class balanced focal loss and get 79.48% accuracy. Compared with the baseline softmax I ran(77.47%), class balanced focal loss has just 2% gain in performance.

[richardaecn]

Could you please also run the class balanced focal loss? I also ran the class balanced focal loss and get 79.48% accuracy. Compared with the baseline softmax I ran(77.47%), class balanced focal loss has just 2% gain in performance.

Sure. I will re-ran and update to you soon. For the softmax baseline you got, can you also re-run it multiple times (using first 10 lines in cifar_im_trainval.sh) and see if it's consistently achieving that high accuracy? On my side, the softmax baseline is usually 75-76%.

[hbsz123]

ok. I will rerun for several times. Thanks for your patientce.

------------------ Original ------------------ From: Yin Cui notifications@github.com Date: Wed,Jul 31,2019 3:45 PM To: richardaecn/class-balanced-loss class-balanced-loss@noreply.github.com Cc: hbsz 346738732@qq.com, Mention mention@noreply.github.com Subject: Re: [richardaecn/class-balanced-loss] baseline for long-tail cifar10 is 77.47% (#5)

Could you please also run the class balanced focal loss? I also ran the class balanced focal loss and get 79.48% accuracy. Compared with the baseline softmax I ran(77.47%), class balanced focal loss has just 2% gain in performance.

Sure. I will re-ran and update to you soon. For the softmax baseline you got, can you also re-ran it multiple times (using first 10 lines in cifar_im_trainval.sh) and see if it's consistently achieving that high accuracy? On my side, the softmax baseline is usually 75-76%.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[richardaecn]

Could you please also run the class balanced focal loss? I also ran the class balanced focal loss and get 79.48% accuracy. Compared with the baseline softmax I ran(77.47%), class balanced focal loss has just 2% gain in performance.

I got 79.42% with class-balanced focal loss (gamma = 1.0 and beta = 0.9999). I also updated README with tensorboard results.

[hbsz123]

OK, thanks. The baseline I ran with vanilla softmax is almost always above 76.5%.

[hbsz123]

By the way, did you train iNaturalist2018 from scratch in your paper? Why not train with ImageNet pretrained model?

[richardaecn]

Yes, we train iNat from scratch. We did not fine-tune because iNat is large enough for training from scratch and ImageNet data is biased so we want to exclude the influence of ImageNet data. Another reason is that we want to use the same standard training strategy with ImageNet training.

[hbsz123]

Thanks for your reply. The theory of effictive number in your paper is very impressive. Do you have wechat？I have some other questions to ask about and I will close this issue.

------------------ Original ------------------ From: Yin Cui notifications@github.com Date: Fri,Aug 2,2019 0:16 PM To: richardaecn/class-balanced-loss class-balanced-loss@noreply.github.com Cc: hbsz 346738732@qq.com, Mention mention@noreply.github.com Subject: Re: [richardaecn/class-balanced-loss] baseline for long-tail cifar10 is 77.47% (#5)

Yes, we train iNat from scratch. We did not fine-tune because iNat is large enough for training from scratch and ImageNet data is biased so we want to exclude the influence of ImageNet data. Another reason is that we want to use the same standard training strategy with ImageNet training.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.