search

anniesch / jtt

Code for "Just Train Twice: Improving Group Robustness without Training Group Information"
66 stars 16 forks source link

Results in model_outputs/output_test_epoch_{epoch_num}.csv do not agree with log.txt #9

Closed kiranchari closed 1 year ago

kiranchari commented 1 year ago

When I manually computed the group accuracies using the predicted outputs in model_outputs/output_testepoch{epoch_num}.csv, they do not agree with what is reported in log.txt.

For CelebA (ERM), the reported test group acc (Blond_hair and Male) is 50% in log.txt whereas when I manually compute the accuracy for this group using model_outputs/output_testepoch{epoch_num}.csv, it is ~95%.

Any reason why these might differ?

kiranchari commented 1 year ago

I realized that the "indices_Noneepoch{epoch}_val" column in "model_outputs/output_testepoch{epoch_num}.csv" is 0-indexed while the filenames in celebA/data/metadata.csv are 1-indexed (they begin with 000001.jpg). This lead to the discrepancy and accounting for this resolved the issue.