ISC Dataset Results Reproducibility

[rahulv54]

Hi, I trained the ISC model a couple of times, but my validation AUROC is only around 2.8. Am I doing something wrong? Training seems to be fine mostly.

[khaledsaab]

Hi Rahul, happy to help debug the issue. A couple questions:

• what is the command you are running? For example, when I run python -m train train.method=erm train.model_type=resnet50 dataset.source=isic I see a validation AUROC of ~0.77 after a single epoch.
• Did you download the data from here https://github.com/alceubissoto/debiasing-skin?
• Perhaps as a sanity check, can you print out the samples in your train, validation, and test dataloaders? Along with the class balances?

[rahulv54]

Khaled, Thanks for the prompt response.

It works perfectly well on validation dataset. (because of unchanged correlations?)

The numbers that I mentioned are from the test dataset, which corresponds to robust AUROC in this case.

Here is the config I passed as an argument to train.py:

model: model_name: resnet arch: resnet50 dropout: 0 pretrained: True resume_ckpt: False

train: seed: 1 # seeds for isic need to be in [1,5] model_type: "resunet" # ["resnet50, resunet"] method: "erm" # ["erm", "seg"] binary_weight: 0 epochs: 100 batch_size: 16 lr: 5e-4 wd: 0 valid_split: val model_id: null

dataset: source: "isic" # options: {"cxr_p", "isic"} sample_ratio: 1 num_workers: 4 id_column: "id" input_column: "input" augmentation: True

wandb: project: domino group: '' log_model: False

I tried the experiment with resnet50 too with similar results.

[rahulv54]

Aside, thanks for sharing the code. I learnt so much about various new libraries too!

[khaledsaab]

Hey Rahul, apologies for the delayed response. Glad you enjoyed the code!

I found that the following can make a good difference:

• train for less epochs
• increase the weight decay

For example, maybe try this: python -m train train.method=erm train.model_type=resnet50 dataset.source=isic train.epochs=5 train.wd=0.01. That should get you to the 30's on the robust AUC for the test.

Also, I am assuming you are using the notebook isic_evaluation.ipynb to do the final evaluation? Just want to make sure we're comparing similar code.

I think the main message here is that just using the image-level binary labels performs significantly worse versus supervising with pixel-level labels (i.e., segmentation). Let me know if you are able to reproduce this main result, where doing segmentation improves robust AUROC to the high 70s range.