jortfal
commented
14 hours ago Hi!
I can understand that the distribution of resources is automatically managed by LM Studio.
But I think it is necessary to be able to select on which GPU to load the model and run it.
I just connected a second GPU to my PC. It is an 8GB Asus 3070Ti to my PC for use with my monitors and my system in general. Since I have a 24GB Nvidia RTX 3090 that I want to use exclusively with LM Studio.
Is it possible to somehow tell LM Studio to only use one GPU? If not currently possible, will it be possible in the next version of LM Studio?
Translated with DeepL.com (free version)
I'm trying to get configured a 6x GPU setup for offloading across available GPUs, but I can't see anywhere on the app any option for selecting GPUs nor multi-GPU options. I noticed that LMS 0.3.5 has detected all the GPUs at the Hardware Info > GPUs section at System Resources of the App Settings, and I can also see all the VRAM combined 72GB (6 * GPU-12GB-VRAM) at Memory Capacity section with the Vulkan engine selected, but there are not any multi-GPU configuration options over there.
In previous versions of LMS, we were able to see the GPU preferences pane in Advance Settings (Right sidebar), but I took a long time off, and when returned back to my tinkering with LMS and my setup, I decided to upgrade to the latest version (0.3.5), and I can't find any multi-GPU options anymore. So, even furthermore, during my research my GPUs gets VRAM consumed when loading very large models, and their utilization and memory clock speeds get a boost/peak at the inference time.
So, the big question I have, does this means that the resource distribution is handled automatically by LMS, and there is no need for LMS's Users/developers to manage offloading across multiple GPUs anymore?
💭 🤔
Hardware:
OS: Windows 11 (23H2) CPU: Intel Core i3-10100 RAM: Crucial PRO 32GB Storage: Crucial BX SATA SSD 250 GB GPUs: 6x - AMD XFX Merc RX6700XT