How to choose GPU with AllGPT binding.

[ediweber]

Hello.

First things first: Thank you for creating lollms, it is simply spoken wonderful!

Unfortunately I suffer with the AllGPT binding. I cannot load my GGUF models, output is:

Lollms webui version : 6.8 Listing all extensions Listing all personalities Listing models Loading discussion for client mrQYbMGCR0JO3rJSAAAB Checking for updates from E:\lollms-webui update availability: True Listing all extensions

• Retreiving binding settings Requested updating of setting model_name to Naberius-7B-GGUF Changing model to: Naberius-7B-GGUF Loading model:naberius-7b.Q8_0.gguf Folder:E:\loldata\models\gguf Error allocating memory ErrorOutOfDeviceMemory error loading model: Error allocating vulkan memory. llama_load_model_from_file: failed to load model LLAMA ERROR: failed to load model from E:\\loldata\\models\\gguf\naberius-7b.Q8_0.gguf Couldn't load model: [Model not loaded] Traceback (most recent call last): File "E:\lollms-webui\app.py", line 914, in update_setting self.model = self.binding.build_model() File "E:\lollms-webui\zoos\bindings_zoo\gpt_4all__init__.py", line 120, in build_model self.model.model.set_thread_count(self.binding_config.n_threads) File "C:\Users\eduar\anaconda3\envs\lol\lib\site-packages\gpt4all\pyllmodel.py", line 274, in set_thread_count raise Exception("Model not loaded") Exception: Model not loaded

I suspect that it tries to use my APU (Ryzen 5600) @ 4GB instead of my Nvidia 3060 - 12 GB, is there a simple way to force it to use the Nvidia card.

Will try with smaller models. In fact it would be nice to have a small one running on the APU and a large one on the dedicated GPU card.

[ediweber]

sorry. I accidently posted this in lollms while this is an lollms-webui issue

[ParisNeo]

Well, in the parameters of gpt4all yo ucan select GPU (cuda)