search

guidance-ai / guidance

A guidance language for controlling large language models.
MIT License
18.73k stars 1.03k forks source link

Multi-GPUs usage #525

Open akkasi opened 9 months ago

akkasi commented 9 months ago

Hi, Could you please let me know how I can use the guidance with multi-gpu settings? I tried models.transformers(modelname, device-map='auto'), but checking with nvidia-smi during the inference time, I see that the GPUs are not used.

slundberg commented 9 months ago

Hmm. It should just pass those params to the underlying transformers model. I have a single GPU box right now though so it is hard for me to test. If you find more details I can comment though.

aadityabhatia commented 7 months ago

device_map="auto" works for me. That underscore might be the issue.