bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
771 stars 201 forks source link

Port BLEU computation #29

Closed Muennighoff closed 1 year ago

Muennighoff commented 1 year ago

Results are now very close to Ours: L-R single 16.05 from Table 2:

accelerate launch main.py --model facebook/incoder-6B --tasks codexglue_code_to_text-python-left --batch_size 1 --limit 50 --do_sample False --save_references True --output_path codexglue_code_to_text-python-left_incoder_greedy_500.json --max_length_generation 128
{
  "codexglue_code_to_text-python-left": 15.590834255518017,
  "config": {
    "model": "facebook/incoder-6B"
  }
}