(LM Studio)Failed to load LLM engine from path (6750XT/Windows)

[kaattaalan]

Tried following the wiki https://github.com/likelovewant/ROCmLibs-for-gfx1103-AMD780M-APU/wiki/Unlock-LM-Studio-on-Any-AMD-GPU-with-ROCm-Guide#using-amd-graphics-cards-with-lm-studio

Copied the files, restarted LMS. The GPU (gfx1031) is still showing up as unsupported

When tried loading the model anyway, got the following error in LM studio :

Failed to load LLM engine from path: C:\Users\Username.cache\lm-studio\extensions\backends\win-llama-rocm-0.2.31\llm_engine_rocm.node. The specified module could not be found. \?\C:\Users\Username.cache\lm-studio\extensions\backends\win-llama-rocm-0.2.31\llm_engine_rocm.node

[likelovewant]

Compatibility Issue with "compatible" Keyword

The compatible keyword might not be working currently due to recent updates from LM Studios. This issue is expected to be resolved in the LM studio next release.

Workaround for Now:

While waiting for the fix, you can still make it work by following these steps:

1. Ignore the compatible keyword:
2. Use ROCm 5.7 Libraries: Ensure your rocm extension uses the ROCm 5.7 libraries as LM Studio currently only supports this version.
3. Replace Llama and GGML DLLs: Download and replace llama.dll and ggml.dll from Ollama for amd repository to your lmstudio extension works .

Choosing the Correct Ollama Version:

• Access the developer mode (in lm studio ) and select the ROCm backend.
• Carefully choose the Ollama version that supports ROCm 5.7. Note that the latest Ollama for AMD only supports ROCm 6.1.2, so you'll need to use an older version compatible with ROCm 5.7. By the way , the 5.7 lastest ollama version will be included in the next release 7z files also . @kaattaalan

[kaattaalan]

It works !!

Thanks a lot for the reply. I tried your Workaround and the models are loading and generating now. (I am getting GGGG output, but fixed it by turning Flash attention on)

But for some reason, it only works for the version 1.1.5 of the extension. If I try updating it to 1.1.10 (to see whether GGG output will be fixed) and do the same steps, it won't work anymore : I installed the extension by following the guide : https://github.com/lmstudio-ai/configs/blob/main/Extension-Pack-Instructions.md

Here's what I did (for reference)

1. I downloaded v0.3.6 of ollama-for-amd
2. Copied ollama-windows-amd64\ollama_runners\rocm_v5.7 llama and ggml dlls to lm-studio\extensions\backends\llama.cpp-win-x86_64-amd-rocm-avx2-1.1.10 replacing the existing llama.dll
3. Then I downloaded v0.5.7of ROCmLibs-for-gfx1103-AMD780M-APU (https://github.com/likelovewant/ROCmLibs-for-gfx1103-AMD780M-APU/releases/download/v0.5.7/rocm.gfx1031.for.hip.sdk.5.7.optimized.with.little.wu.logic.and.I8II.support.7z)
4. I then extracted the rocblas.dll and library contents to lm-studio\extensions\backends\vendor\win-llama-rocm-vendor-v2
5. Restarted LM studio, chose the correct runtime :
6. Then I was able to load the model (with flash attention on) without errors, even though it shows up as incompatible.

[likelovewant]

1.1.10 need update llama.dll and ggml.dll from ollama build with rocm5.7 ,however , the files from ollama for amd build with 6.1.2 . The possible Solution would be build ollama yourself with rocm 5.7 and grab the files from the build or wait for the next release which may include 5.7 .

[kaattaalan]

Thanks a ton for the reply. I don't mind waiting for the next build, while I will be experimenting with ollama for some days. Thanks for all the hard work.

[likelovewant]

had update the ollama-windows-amd64-rocm-5.7.7z ,are able to work with 1.1.10 entension for LM Studio. Feel free to test it .

[kaattaalan]

Just tested ollama 0.3.11(5.7.7 build) with rocm extension 1.1.10. Models are not loading. The only combo that seems to work is 0.3.6 with 1.1.5.

[likelovewant]

it's a bit tricky . you also need to re-edit information and replace files as steps before .if you missing one of the steps or use the libs from rocm 6.1.2 , it won't load .sometime , latest drivers help.