IndexError: list index out of range

[super-buster]

Hi, Lisali. I failed to use GPT-2, which trained in prefix tuning mode on webnlg mode, to evaluate valid dataset. I use the scripts you provide here. The detailed information shows in the belowing.

Traceback (most recent call last): File "evaluation.py", line 356, in <module> read_participant(team, sys.argv[1]) File "evaluation.py", line 250, in read_participant output_reduced = [output[i-1] for i in sorted(entry_ids)] File "evaluation.py", line 250, in <listcomp> output_reduced = [output[i-1] for i in sorted(entry_ids)] IndexError: list index out of range

the command is bash evaluation/run_eval_on_webnlg.sh ~/PrefixTuning/transformers/examples/text-generation/webNLG_results2/webnlgprefixtune_y_5_act_cat_b=5-e=5_d=0.0_u=no_lr=5e-05_w=0.0_s=101_r=n_m=512_o=1_o=1_valid_beam webnlgprefixtune_y_5_act_cat_b=5-e=5_d=0.0_u=no_lr=5e-05_w=0.0_s=101_r=n_m=512_o=1_o=1_valid_beam >> ~/PrefixTuning/transformers/examples/text-generation/webNLG_results2/webnlgprefixtune_y_5_act_cat_b=5-e=5_d=0.0_u=no_lr=5e-05_w=0.0_s=101_r=n_m=512_o=1_o=1_valid_beam_eval

I have observed "sorted(entry_ids)" starts with element 0, and it's length can not match output. I read the code and find output is just the file that I provide in the command line and assure it's justification. Do I miss something? Thank you very much.

[XiangLi1999]

Hi,

I think you need to manually change the script a bit, so that it has the right validation reference? maybe one things to check first is to see if the line count in *valid_beam_eval is equal to the line count in the reference file.

[goodcode309]

it seems like the script can‘t evaluate on valid dataset, because the "goldfile" only contains "testdata_with_lex.xml", so you can just replace "valid" with "test" in the command.