cshizhe / VLN-HAMT

Official implementation of History Aware Multimodal Transformer for Vision-and-Language Navigation (NeurIPS'21).
MIT License
99 stars 12 forks source link

The test set result output is missing during the inference phase #18

Closed jj023721 closed 2 weeks ago

jj023721 commented 2 weeks ago

The code where the problem occurs:

I ran the inference script run_r2r.sh, but did not see any output about the test set. I checked the code and found that Line257 in r2r/main.py does not process the result output of processing the test set.

        if default_gpu:
            if 'test' not in env_name:
                score_summary, _ = env.eval_metrics(preds)
                loss_str = "Env name: %s" % env_name
                for metric, val in score_summary.items():
                    loss_str += ', %s: %.2f' % (metric, val)
                write_to_record_file(loss_str+'\n', record_file)

After modifying the if condition, the output of the test set results is abnormal

val_train_seen cost time: 14.04s
eval 1501 predictions
Env name: val_train_seen, steps: 5.19, lengths: 9.98, nav_error: 0.94, oracle_error: 0.69, sr: 91.41, oracle_sr: 93.54, spl: 89.43, nDTW: 90.33, SDTW: 86.53, CLS: 89.04

val_seen cost time: 9.92s
eval 1021 predictions
Env name: val_seen, steps: 5.58, lengths: 11.25, nav_error: 2.69, oracle_error: 1.80, sr: 74.14, oracle_sr: 81.39, spl: 71.06, nDTW: 77.48, SDTW: 67.53, CLS: 76.25

val_unseen cost time: 22.57s
eval 2349 predictions
Env name: val_unseen, steps: 6.30, lengths: 12.30, nav_error: 3.77, oracle_error: 2.42, sr: 64.41, oracle_sr: 72.46, spl: 58.73, nDTW: 66.58, SDTW: 55.04, CLS: 65.55

test cost time: 42.45s
eval 4173 predictions
Env name: test, steps: 6.47, lengths: 13.03, nav_error: 8.85, oracle_error: 0.00, sr: 4.46, oracle_sr: 100.00, spl: 0.00, nDTW: 0.19, SDTW: 0.02, CLS: 0.00