search

likelovewant / ROCmLibs-for-gfx1103-AMD780M-APU

ROCm Library Files for gfx1103 and update with others arches based on AMD GPUs for use in Windows.
GNU General Public License v3.0
137 stars 14 forks source link

Weird Output in LMS 0.3.5 with `win-llama-rocm-0.2.31` #16

Open gchriswill opened 6 days ago

gchriswill commented 6 days ago

Screenshot 2024-11-10 000712

Screenshot 2024-11-10 020119

Screenshot 2024-11-10 020202

Software Stack:

Hardware Stack:

OS: Windows 11 (23H2) CPU: Intel Core i3-10100 RAM: Crucial PRO 32GB Storage: Crucial BX SATA SSD 250 GB GPUs: 6x - AMD XFX Merc RX6700XT

Being 2 days going back and forth trying to have LM Studio + ROCm + 6x 6700XT. I got to the point of LM Studio recognizing the GPUs (Proven at third screenshot of LMS System Resources with 72GB of VRAM), and being able to get compatible status for the llama-ROCm runtime (Proven at second screenshot), and even loading different models. But the responses seems to be machine code or some sort of combination/mix from text response and code.

@likelovewant, Any suggestions/ideas of what might be the issue?

Thanks in advance 🙏

likelovewant commented 6 days ago

Try ROCmLibs for 6.1.2, LM studio had updated it , if still talking no sense , try different ollama for amd version. @gchriswill

gchriswill commented 6 days ago

@likelovewant Ok, will try the following Software Stack:

I will delete and reset everything to have a clean and fresh start to triage the stack with the updated versions, and post back the results in this issue thread. 👍

gchriswill commented 6 days ago

@likelovewant Also, just for informational purposes here is another interesting scenario from my setup in a ticket I submitted to LMS repo. https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/194 (Unliked due to is unrelated)

likelovewant commented 6 days ago

@likelovewant Ok, will try the following Software Stack:

  • LMS 0.3.5 + ROCm Llama.cpp 1.1.10 (from LMS runtimes)
  • HIP SDK 6.1.0
  • ROCmLibs for 6.1.2
  • Ollama For AMD 0.4.1
  • AMD Radeon Software 24.10.1

I will delete and reset everything to have a clean and fresh start to triage the stack with the updated versions, and post back the results in this issue thread. 👍

1,ROCmLibs for 6.1.2 don't work withOllama For AMD 0.4.1as it was build with Rocm5.7. and there is an breaking change in ollama 0.4.0 and then after . try earlier version . test works withv0.3.14. simply rename ollamaggml.dllto ggml_llamacpp.dll and replace the dll files in LMS0.3.5 and ROCmlibs.

2,* HIP SDK 6.1.0 is not a necessary . As the ROCm libs direct talk to AMD drivers ,ie AMD Radeon Software.

3, Make sure to read the release notes . some build rocm 5.7 ,others build with 6.1. need compitable rocmbs . however ,those trick are changing as the new update in lmstudio or ollama . it's may break anytime.

  1. For some models , need to change the prompts presets in LMS otherwise ,it will reply your gibberish
likelovewant commented 6 days ago

@likelovewant Also, just for informational purposes here is another interesting scenario from my setup in a ticket I submitted to LMS repo. https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/194 (Unliked due to is unrelated)

In ollama , there can be mannuly set gpu_num offload , lmstudio set all depend on their update .

@gchriswill