search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
698 stars 180 forks source link

Some questions about APPS #240

Closed virt9 closed 3 weeks ago

virt9 commented 1 month ago

hello! i have some problems when evaluating on APPS. firstly ,I cant find any one-shot examples on corresponding generation code like follows: and i wander how should i use the locally download metrics without downloading "apps_metrics"? thanks a lot if anyone have some ideas !!!

loubnabnl commented 3 weeks ago

Hi, I'm not sure I understand, can you provide the code you used to run the evaluation? and wdym by

without downloading "apps_metrics"