search

huangd1999 / AgentCoder

This Repo is the official implementation of AgentCoder and AgentCoder+.
122 stars 19 forks source link

Error when running test_executor_humaneval #1

Open HamedTaherkhani opened 2 months ago

HamedTaherkhani commented 2 months ago

Hi, I am getting this error:

Traceback (most recent call last): File "AgentCoder/test_executor_humaneval.py", line 323, in <module> test_report(dataset,lg) File "AgentCoder/test_executor_humaneval.py", line 290, in test_report result = check_correctness(dataset[i]["task_id"],dataset[i],lg,5,"./tmp") File "AgentCoder/./CodeGeeX/codegeex/benchmark/execution.py", line 378, in check_correctness "test_code" : sample["test_code"], KeyError: 'test_code'

Can you please fix this?

huangd1999 commented 2 months ago

Hi, I have checked the source code provided by CodeGeeX. I think this problem can be addressed by directly replacing "test_code" with "full_code".

By the way, we have updated human eval files since we observed that the results will be hard to control with massive prompt instruction. In the current version, the pass@1 of humaneval for ChatGPT increases from 79.9% to 80.5%. You can pull the repo for the new version.