Discrepancies in reproducing results

[ADevillers]

Description: I encountered discrepancies while attempting to reproduce the results presented in your paper titled "Learning De-biased Representations with Biased Representations." Despite following the provided methodology and using the suggested code and parameters on GitHub, the obtained results differ from what was reported in the paper.

Expected Results:

• Rubi 0.99:

  • Run 1: ~0.936
  • Run 2: ~0.936
  • Run 3: ~0.936

• ReBias 0.999:

  • Run 1: ~0.227
  • Run 2: ~0.227
  • Run 3: ~0.227

Actual Results:

• Rubi 0.99:

  • Run 1: 0.8578
  • Run 2: 0.8666
  • Run 3: 0.8578

• ReBias 0.999:

  • Run 1: 0.2419
  • Run 2: 0.244
  • Run 3: 0.253

Steps to Reproduce:

1. Clone the repository and install requirements:

   git clone https://github.com/clovaai/rebias.git
   cd rebias
   pip install -r requirements.txt

2. For Rubi 0.99:

   python main_biased_mnist.py --root data/ --train_correlation 0.99 --outer_criterion RUBi --g_lambda_inner 0

3. For ReBias 0.999:

   python main_biased_mnist.py --root data/ --train_correlation 0.999

I would greatly appreciate your assistance in understanding the reasons behind these inconsistencies.

Thank you for your time, and I look forward to your response.

Sincerely, Alexandre DEVILLERS

[SanghyukChun]

Hi, did you check the PyTorch version as well? What is the output of this command? pip freeze | grep torch

[SanghyukChun]

Okay, I just found that torch==1.1.0 is no longer available at pip. Check this document for installing v1.1.0 https://pytorch.org/get-started/previous-versions/#v110

I am now checking whether the performances are reproduced under torch 1.4.0 and torchvision 0.5.0

[SanghyukChun]

Using torch==1.4.0, I got the following results, where (number) denotes the reported numbers in the paper. (not repeated 3 times, but only measured once)

rho	RUBi	ReBias
0.999	10.0 (13.7)	24.4 (22.7)
0.997	15.0 (43.0)	64.2 (64.2)
0.995	61.2 (90.4)	76.6 (76.0)
0.990	84.9 (93.6)	87.1 (88.1)

It seems that the RUBi implementation is not compatible with recent torch packages. The ReBias result looks less critical, and just an okay result to me.

I am not willing to update the RUBi implementation due to the following reasons:

• This is an old codebase. It already took 4 years since I first implemented the code.
• I don't have a torch 1.1.0 compatible device anymore (CUDA 10.0); therefore, it is impossible to compare the behavior of the code by torch 1.1.0 and recent torch versions.
• Finally, I am not actively working on this topic anymore. I don't have enough time to dig into the issue.
• This codebase is for ReBias, not RUBi. If ReBias implementation is problematic, then I maybe can put my effort into this issue, but it seems that the problem is only for RUBi.