bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
771 stars 201 forks source link

Change bool arguments format #43

Closed loubnabnl closed 1 year ago

loubnabnl commented 1 year ago

Use store_true and store_false actions for boolean arguments to avoid confusions with boolean type. That was happening to allow_code_execution where any value (e.g False) was interpreted as a string resulting in True (we reformat the other arguments for consistency).

now one should execute:

accelerate launch  main.py \
  --model <MODEL_NAME> \
  --tasks <TASK_NAME> \
  --allow_code_execution \
  --save_generations

instead of

accelerate launch  main.py \
  --model <MODEL_NAME> \
  --tasks <TASK_NAME> \
  --allow_code_execution True \
  --save_generations True

cc @infinitylogesh @Muennighoff