Wrong evaluation reply using ChatGPT

[anirudh-chakravarthy]

Hi,

When I use my output.json and the repo's test_eval.json, it worked the first two times. However, now I see ChatGPT replies such as:

I would rate your answer as 10.

which leads to the following error:

evaluation/0 [0]:  File "/workspace/evaluation.py", line 194, in <module>
evaluation/0 [0]:    output = evaluation.evaluation()
evaluation/0 [0]:  File "/workspace/evaluation.py", line 153, in evaluation
evaluation/0 [0]:    scores["match"] = self.eval_match()
evaluation/0 [0]:  File "/workspace/evaluation.py", line 71, in eval_match
evaluation/0 [0]:    outs2 = self.eval_chatGPT(self.match["GPT"])
evaluation/0 [0]:  File "/workspace/evaluation.py", line 46, in eval_chatGPT
evaluation/0 [0]:    scores = list(map(float, scores))
evaluation/0 [0]:ValueError: could not convert string to float: 'I would rate your answer as 10.'

Based on previous issues https://github.com/OpenDriveLab/DriveLM/issues/92, https://github.com/OpenDriveLab/DriveLM/issues/53, it seems like this should have been fixed, but I'm still facing this issue. Can someone help me out?

[ChonghaoSima]

Thank you for your interest. Are you running eval on your local env or our test server?