search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
781 stars 208 forks source link

Support for StudentEval Dataset (Again) #198

Open guanqun-yang opened 7 months ago

guanqun-yang commented 7 months ago

@arjunguha It seems that #88 is not merged into the main branch but is somehow closed. Running the code taken from the HuggingFace data card returns errors:

python3 main.py --model bigcode/gpt_bigcode-santacoder --tasks studenteval --max_length_generation 512 --n_samples 20 --batch_size 20 --precision bf16 --allow_code_execution

Is there any way I could run the evaluation on the StudentEval dataset any time soon?

arjunguha commented 7 months ago

whoops. mistake. let me try to merge it in this time. i can probably get it running in a day or so