search

lbaermann / qaego4d

Code and Dataset for the CVPRW Paper "Where did I leave my keys? — Episodic-Memory-Based Question Answering on Egocentric Videos"
19 stars 0 forks source link

Try to use eval.py #5

Closed Deaddawn closed 3 months ago

Deaddawn commented 5 months ago

Hi, I am trying to use eval.py, for gold_answers and eval_file, I pass answers.json for a test, but got this error:

raise TypeError(f'{prefix}: {err_msg}')

TypeError: BLEU: refs should be a sequence of sequence of strings.

Deaddawn commented 5 months ago

Hi, I am trying to use eval.py, for gold_answers and eval_file, I pass answers.json for a test, but got this error:

raise TypeError(f'{prefix}: {err_msg}')

TypeError: BLEU: refs should be a sequence of sequence of strings.

A lot of bugs going on here, suggest take a look

lbaermann commented 5 months ago

If you take a look at https://github.com/lbaermann/qaego4d/blob/bc1e29d3138a7db4ba35e19d9c1f05302e0c34c3/eval/eval.py#L123 how the answers json file is handled, the script directly expects a mapping from sample id to answer. So you need to extract one of the split dictionaries (train, val, test) in the file answers.json as provided in the dataset, or you "flatten" that file to contain all samples from all splits at once.

If this does not help, can you provide more details (stack trace, input etc)?