bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
710 stars 183 forks source link

how to use --instruction_tokens? #103

Closed wwngh1233 closed 11 months ago

loubnabnl commented 1 year ago

This argument is only used for InstructHumanEval benchmark and its use is explained in this documentation. For example:

accelerate launch  main.py \
  --model <MODEL_NAME> \
  --max_length_generation <MAX_LENGTH> \
  --tasks instruction-humaneval \
  --instruction_tokens <user_token>,<end_token>,<assistant_token>\
  --temperature 0.2 \
  --n_samples 50 \
  --batch_size 50 \
  --allow_code_execution