search

anniesch / jtt

Code for "Just Train Twice: Improving Group Robustness without Training Group Information"
66 stars 16 forks source link

Some errors in the sample commands for Multi-NLI #7

Closed LKChenLK closed 2 years ago

LKChenLK commented 2 years ago
  1. On line 2 of the sample command for MNLI, ERM_upweight_0_epochs_5_lr_2e-05_weight_decay_0.0 is generated from line 1 instead of ERM_upweight_0_epochs_5_lr_2e-05_weight_decay_0
  2. The --folder_name argument on line 3 should probably be ERM_upweight_0_epochs_5_lr_2e-05_weight_decay_0.0 instead of ERM_upweight_0_epochs_5_lr_2e-05_weight_decay_0.0_nobert , the latter wasn't generated and doesn't exist.
  3. On line 4, JTT_upweight_0_epochs_5_lr_2e-05_weight_decay_0 should probably be JTT_upweight_0_epochs_5_lr_2e-05_weight_decay_0.1.
  4. The default metadata file name is metadata.csv, so using metadata_random.csv as instructed without changing input flags (like in the sample commands) results in FileNotFoundError.

edit: removed markdown in inline code.

anniesch commented 2 years ago

Thanks for the note! I've updated the README, let me know if there are any other issues!

LKChenLK commented 2 years ago

Hello, Thanks for the update! There is another difference between the generated command and the sample command: for line 4 fo the MNLI sample commands, instead of JTT_upweight_0_epochs_5_lr_2e-05_weight_decay_0.1 , it should probably be JTT_upweight_4_epochs_5_lr_1e-05_weight_decay_0.1 , since in the previous command --lr is set to 1e-5 .

Also: I mistyped the upweight factor as 0 for my 3rd point in my previous comment (sorry!) In the current sample commands it is set to 4; however the updated version of the paper reported that upweight factor of 6 gives the best results. Maybe you would want to change the upweight factor in README to 6 as well?