search

bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
709 stars 183 forks source link

code problem #139

Closed ALLISWELL8 closed 8 months ago

ALLISWELL8 commented 10 months ago

How to specify multiple A100 models with larger parameters for this project when one A100 graphics memory is insufficient？

loubnabnl commented 10 months ago

You can ask accelerate to use multiple GPUs when doing accelerate config , if your model doesn't fit in memory you can add the flag --max_memory_per_gpu "auto" and your model will be sharded on multiple GPUs. But ca you provide more details on which model are you running and how much RAM you have in your A100, most medium-sized models should fit with half-precision and a small batch size.