search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
825 stars 219 forks source link

Run 70b evaluation. #235

Closed icoderzqliu closed 6 months ago

icoderzqliu commented 6 months ago

Hello, when I run the evaluation of the 70b model, I used 8*80G GPUs, but it still oom. How can I evaluate the the 70b model?

OTTO-OTO commented 2 months ago

hello, I have met the same problem, may I ask how you solve it?

loubnabnl commented 2 months ago

You can set --max_memory_per_gpu flag to auto and it will shard the model on the available gpus. You can also reduce the batch size to 1, and use bf16 --precision bf16 and if it still OOMs, you can try reducingmax_length_generation