KeyError: 'env5_in_acc' | DomainNet experiment

[MohammadJavadD]

While running "domainbed.scripts.collect_results", I am getting the "KeyError: 'env5_in_acc" error. Also, I notice the "DomainNet experiment" just print env0-env4 while it supposed to print env5 as well. I was wondering if you could help me to fix it.

The error: Traceback (most recent call last): File "./anaconda3/envs/wavenet3/lib/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "./anaconda3/envs/wavenet3/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/./DomainBed/domainbed/scripts/collect_results.py", line 187, in <module> print_results_tables(records, selection_method, args.latex) File "/./DomainBed/domainbed/scripts/collect_results.py", line 91, in print_results_tables grouped_records = get_grouped_records(records).map(lambda group: File "/./DomainBed/domainbed/lib/query.py", line 111, in map return Q([fn(x) for x in self._list]) File "/./DomainBed/domainbed/lib/query.py", line 111, in <listcomp> return Q([fn(x) for x in self._list]) File "/./DomainBed/domainbed/scripts/collect_results.py", line 92, in <lambda> { **group, 'sweep_acc': selection_method.sweep_acc(group['records']) } File "/./DomainBed/domainbed/model_selection.py", line 34, in sweep_acc .map(lambda _, run_records: self.run_acc(run_records)) File "/./DomainBed/domainbed/lib/query.py", line 109, in map return Q([fn(*x) for x in self._list]) File "/./DomainBed/domainbed/lib/query.py", line 109, in <listcomp> return Q([fn(*x) for x in self._list]) File "/./DomainBed/domainbed/model_selection.py", line 34, in <lambda> .map(lambda _, run_records: self.run_acc(run_records)) File "/./DomainBed/domainbed/model_selection.py", line 89, in run_acc return test_records.map(self._step_acc).argmax('val_acc') File "/./DomainBed/domainbed/lib/query.py", line 111, in map return Q([fn(x) for x in self._list]) File "/./DomainBed/domainbed/lib/query.py", line 111, in <listcomp> return Q([fn(x) for x in self._list]) File "/./DomainBed/domainbed/model_selection.py", line 81, in _step_acc 'test_acc': record[test_in_acc_key] KeyError: 'env5_in_acc'

DomainNet experiment output: env0_in_acc env0_out_acc env1_in_acc env1_out_acc env2_in_acc env2_out_acc env3_in_acc env3_out_acc env4_in_acc env4_out_acc epoch loss step step_time 0.0028049034 0.0023896104 0.0046749346 0.0036818138 0.0031480809 0.0030443507 0.0027463768 0.0024927536 0.0041326210 0.0050911376 0.0000000000 5.1279506683 0 0.6886174679

[lopezpaz]

That's interesting, attaching my command and output where I can see all environment statistics:

python -m domainbed.scripts.train --data_dir /checkpoint/dlp/datasets --dataset DomainNet --checkpoint_freq 1 --hparams '{"batch_size": 16}'

Args:
        algorithm: ERM
        checkpoint_freq: 1
        data_dir: /checkpoint/dlp/datasets
        dataset: DomainNet
        holdout_fraction: 0.2
        hparams: {"batch_size": 16}
        hparams_seed: 0
        output_dir: train_output
        seed: 0
        steps: None
        test_envs: [0]
        trial_seed: 0
HParams:
        batch_size: 16
        class_balanced: False
        groupdro_eta: 0.01
        irm_lambda: 100.0
        irm_penalty_anneal_iters: 500
        lr: 5e-05
        mixup_alpha: 0.2
        mldg_beta: 1.0
        mlp_depth: 3
        mlp_dropout: 0.0
        mlp_width: 256
        mmd_gamma: 1.0
        resnet_dropout: 0.0
        weight_decay: 0.0
env0_in_acc   env0_out_acc  env1_in_acc   env1_out_acc  env2_in_acc   env2_out_acc  env3_in_acc   env3_out_acc  env4_in_acc   env4_out_acc  env5_in_acc   env5_out_acc  epoch         loss          step          step_time    
0.0017660503  0.0021818182  0.0022042438  0.0018409069  0.0015913376  0.0015221753  0.0024927536  0.0024057971  0.0033030255  0.0026019833  0.0014646583  0.0013743219  0.0000000000  5.8700618744  0             1.6476581097
...

[igul222]

I wasn't able to reproduce this either. Could you post instructions to reproduce?

[ds-giao]

I meet this problem too，do you solve it?