search

CodedotAl / gpt-code-clippy

Full description can be found here: https://discuss.huggingface.co/t/pretrain-gpt-neo-for-open-source-github-copilot-model/7678?u=ncoop57
Apache License 2.0
3.29k stars 220 forks source link

Update Eval Results for HumanEval #69

Open ncoop57 opened 2 years ago

ncoop57 commented 2 years ago

Need to update the HumanEval results due to bug that was originally in our evaluation code and was fixed in this PR #62

neubig commented 2 years ago

I'm interested in this! Are there any directions on how to run the HumanEval evaluation? I looked through the "evaluation" directory briefly but it wasn't immediately obvious what to do.

ncoop57 commented 2 years ago

No, not yet I'm actually creating it right now as there are a few weird dependencies. I'll post here once I have the README up. Would be awesome to get some help from you @neubig on running it as I'm having some weird behavior when running the completions!

neubig commented 2 years ago

Great, I'll try to take a look once there's a bit more doc.

ncoop57 commented 2 years ago

Just uploaded a readme in the evaluation folder that has the steps. Hadn't had the chance to test it but it should hopefully get you able to start evaluating a model

reshinthadithyan commented 2 years ago

I can reproduce the set-up based on the readme.

Just uploaded a readme in the evaluation folder that has the steps. Hadn't had the chance to test it but it should hopefully get you able to start evaluating a model