How to Run With Multiple GPU?

abhinand5 / gptq_for_langchain

A guide about how to use GPTQ models with langchain

GNU General Public License v3.0

40 stars 9 forks source link

How to Run With Multiple GPU? #2

Open yunusemredemirdag opened 1 year ago

yunusemredemirdag commented 1 year ago

Hi, How to run demo.py with multiple gpu?

When I run this code, I take following error:

OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 47.54 GiB total capacity; 20.74 GiB already allocated; 388.25 MiB free; 20.74 GiB reserved in total by 
PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

abhinand5 commented 1 year ago

This should be possible with a few tweaks. I'll try to add support for multiple GPUs by this weekend.