bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
771 stars 201 forks source link

Fix number of copies per task when nsamples is not proportional to batch size #54

Closed loubnabnl closed 1 year ago

loubnabnl commented 1 year ago

If the batch size is not proportional to nsamples the previous formula ncopies = nsamples // batch_size would generate an insufficient number of copies per task to reach nsamples (issue #48 ). We use ceil instead and delete extra completions.