search

CodedotAl / gpt-code-clippy

Full description can be found here: https://discuss.huggingface.co/t/pretrain-gpt-neo-for-open-source-github-copilot-model/7678?u=ncoop57
Apache License 2.0
3.3k stars 224 forks source link

Fix HumanEval evaluation script. #62

Closed HiroakiMikami closed 3 years ago

HiroakiMikami commented 3 years ago

Thank you so much for this great work!

This PR removes the prompts from the human-eval completions task to work human-eval module correctly.

human-eval module concatenates prompt and completion before executing the script (code). Thus, I think we must not include the prompts in the completions.

ncoop57 commented 3 years ago

Thank you so much for this PR and finding this issue! I'll be able to review this PR this weekend to make sure it does not hurt our performance on any of the tasks.