Mistake in collect_results.py with --single_test_envs sweep option

[abalhomaid]

Summary

model_selection.py:90 uses test environment numbers in test_in_acc_key = 'env{}_in_acc'.format(test_env) but if we have two environments: env0 and env1, the error KeyError: 'env3_in_acc' is produced.

Replication steps

• Pick a dataset with two domains
• Run sweep with as few trials as possible like the following
  • python -m domainbed.scripts.sweep launch\ --data_dir=./domainbed/data/\ --output_dir=./my/sweep/\ --algorithms CORAL\ --datasets \ --n_hparams 1\ --n_trials 1\ --seed 0\ --holdout_fraction 0.1\ --command_launcher local\ --skip_confirmation --single_test_envs
• Sweep produces 4 directories under ./my/sweep/<two_domain_dataset>
• Each directory has out.txt with the following accuracies: env0_in_acc env0_out_acc env1_in_acc env1_out_acc
  • test environments are numbered with [0], [1], [2], [3]
• collect_results.py:170 uses model_selection IIDAccuracySelectionMethod
• modelselection.py:90 uses the test environment_ numbers in test_in_acc_key = 'env{}_in_acc'.format(test_env) but we only have env0 and env1

Hence the error KeyError: 'env3_in_acc' is produced.

[abalhomaid]

The mistake is on my side, I specified 4 environments under datasets.py while I trained on two. Closing the issue