search

likelovewant / ollama-for-amd

Get up and running with Llama 3, Mistral, Gemma, and other large language models.by adding more amd gpu support.
https://ollama.com
MIT License
154 stars 13 forks source link

Failing on gfx1032 #26

Closed Astlaan closed 3 hours ago

Astlaan commented 3 hours ago

What is the issue?

My hardware is: CPU: AMD 6800HS GPU: AMD 6700S (Corresponds to gfx1032, from what I can tell)

1) I installed your Ollama for AMD latest release on Windows. 2) I downloaded the 1032 Freemont Dango option from https://github.com/likelovewant/ROCmLibs-for-gfx1103-AMD780M-APU/releases/tag/v0.6.1.2. 3) I extracted and substituted the library folder in AppData\Local\Programs\Ollama\lib\ollama\rocblas 4) I set the debug environment variable with $env:OLLAMA_DEBUG = "1" 5) I run ollama serve, and get:

ollama serve
2024/10/26 14:35:43 routes.go:1170: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\user\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-10-26T14:35:43.527+01:00 level=INFO source=images.go:754 msg="total blobs: 5"
time=2024-10-26T14:35:43.527+01:00 level=INFO source=images.go:761 msg="total unused blobs removed: 0"
time=2024-10-26T14:35:43.528+01:00 level=INFO source=routes.go:1217 msg="Listening on 127.0.0.1:11434 (version 0.3.14-0-gc692dfd)"
time=2024-10-26T14:35:43.529+01:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\user\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe
time=2024-10-26T14:35:43.529+01:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\user\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2024-10-26T14:35:43.529+01:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\user\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2024-10-26T14:35:43.529+01:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\user\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm_v6.1\ollama_llama_server.exe
time=2024-10-26T14:35:43.529+01:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cpu_avx cpu_avx2 rocm_v6.1 cpu]"
time=2024-10-26T14:35:43.529+01:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-10-26T14:35:43.530+01:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-10-26T14:35:43.530+01:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-10-26T14:35:43.530+01:00 level=INFO source=gpu_windows.go:167 msg=packages count=1
time=2024-10-26T14:35:43.531+01:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=8 efficiency=0 threads=16
time=2024-10-26T14:35:43.531+01:00 level=DEBUG source=gpu.go:94 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-10-26T14:35:43.531+01:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=nvml.dll
time=2024-10-26T14:35:43.531+01:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Program Files\\PowerShell\\7\\nvml.dll C:\\Users\\user\\anaconda3\\nvml.dll C:\\Users\\user\\anaconda3\\Library\\mingw-w64\\bin\\nvml.dll C:\\Users\\user\\anaconda3\\Library\\usr\\bin\\nvml.dll C:\\Users\\user\\anaconda3\\Library\\bin\\nvml.dll C:\\Users\\user\\anaconda3\\Scripts\\nvml.dll C:\\Users\\user\\anaconda3\\bin\\nvml.dll C:\\Users\\user\\anaconda3\\condabin\\nvml.dll C:\\WINDOWS\\system32\\nvml.dll C:\\WINDOWS\\nvml.dll C:\\WINDOWS\\System32\\Wbem\\nvml.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\Program Files\\dotnet\\nvml.dll C:\\Program Files (x86)\\FAHClient\\nvml.dll C:\\Program Files\\Git\\cmd\\nvml.dll C:\\Program Files\\CMake\\bin\\nvml.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvml.dll C:\\Program Files\\nodejs\\nvml.dll C:\\WINDOWS\\System32\\OpenSSH\\nvml.dll C:\\Users\\user\\.cargo\\bin\\nvml.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvml.dll C:\\Program Files\\LLVM\\bin\\nvml.dll C:\\Program Files\\GNU\\units\\nvml.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\nvml.dll C:\\Users\\user\\AppData\\Roaming\\npm\\nvml.dll C:\\Windows\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\dotnet\\nvml.dll C:\\Program Files\\Wolfram Research\\WolframScript\\nvml.dll C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\nvml.dll C:\\Program Files\\PowerShell\\7\\nvml.dll C:\\Users\\user\\AppData\\Roaming\\Ruby32-x64\\bin\\nvml.dll C:\\Users\\user\\.cargo\\bin\\nvml.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvml.dll C:\\Program Files\\LLVM\\bin\\nvml.dll C:\\Program Files\\GNU\\units\\nvml.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\nvml.dll C:\\Users\\user\\AppData\\Roaming\\npm\\nvml.dll C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]"
time=2024-10-26T14:35:43.532+01:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths=[]
time=2024-10-26T14:35:43.532+01:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=nvcuda.dll
time=2024-10-26T14:35:43.533+01:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Program Files\\PowerShell\\7\\nvcuda.dll C:\\Users\\user\\anaconda3\\nvcuda.dll C:\\Users\\user\\anaconda3\\Library\\mingw-w64\\bin\\nvcuda.dll C:\\Users\\user\\anaconda3\\Library\\usr\\bin\\nvcuda.dll C:\\Users\\user\\anaconda3\\Library\\bin\\nvcuda.dll C:\\Users\\user\\anaconda3\\Scripts\\nvcuda.dll C:\\Users\\user\\anaconda3\\bin\\nvcuda.dll C:\\Users\\user\\anaconda3\\condabin\\nvcuda.dll C:\\WINDOWS\\system32\\nvcuda.dll C:\\WINDOWS\\nvcuda.dll C:\\WINDOWS\\System32\\Wbem\\nvcuda.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\Program Files\\dotnet\\nvcuda.dll C:\\Program Files (x86)\\FAHClient\\nvcuda.dll C:\\Program Files\\Git\\cmd\\nvcuda.dll C:\\Program Files\\CMake\\bin\\nvcuda.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\nvcuda.dll C:\\Program Files\\nodejs\\nvcuda.dll C:\\WINDOWS\\System32\\OpenSSH\\nvcuda.dll C:\\Users\\user\\.cargo\\bin\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvcuda.dll C:\\Program Files\\LLVM\\bin\\nvcuda.dll C:\\Program Files\\GNU\\units\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\nvcuda.dll C:\\Users\\user\\AppData\\Roaming\\npm\\nvcuda.dll C:\\Windows\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\dotnet\\nvcuda.dll C:\\Program Files\\Wolfram Research\\WolframScript\\nvcuda.dll C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\nvcuda.dll C:\\Program Files\\PowerShell\\7\\nvcuda.dll C:\\Users\\user\\AppData\\Roaming\\Ruby32-x64\\bin\\nvcuda.dll C:\\Users\\user\\.cargo\\bin\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\nvcuda.dll C:\\Program Files\\LLVM\\bin\\nvcuda.dll C:\\Program Files\\GNU\\units\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\nvcuda.dll C:\\Users\\user\\AppData\\Roaming\\npm\\nvcuda.dll C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]"
time=2024-10-26T14:35:43.534+01:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths=[]
time=2024-10-26T14:35:43.534+01:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=cudart64_*.dll
time=2024-10-26T14:35:43.534+01:00 level=DEBUG source=gpu.go:528 msg="gpu library search" globs="[C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Program Files\\PowerShell\\7\\cudart64_*.dll C:\\Users\\user\\anaconda3\\cudart64_*.dll C:\\Users\\user\\anaconda3\\Library\\mingw-w64\\bin\\cudart64_*.dll C:\\Users\\user\\anaconda3\\Library\\usr\\bin\\cudart64_*.dll C:\\Users\\user\\anaconda3\\Library\\bin\\cudart64_*.dll C:\\Users\\user\\anaconda3\\Scripts\\cudart64_*.dll C:\\Users\\user\\anaconda3\\bin\\cudart64_*.dll C:\\Users\\user\\anaconda3\\condabin\\cudart64_*.dll C:\\WINDOWS\\system32\\cudart64_*.dll C:\\WINDOWS\\cudart64_*.dll C:\\WINDOWS\\System32\\Wbem\\cudart64_*.dll C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll C:\\Program Files\\dotnet\\cudart64_*.dll C:\\Program Files (x86)\\FAHClient\\cudart64_*.dll C:\\Program Files\\Git\\cmd\\cudart64_*.dll C:\\Program Files\\CMake\\bin\\cudart64_*.dll C:\\Program Files\\Docker\\Docker\\resources\\bin\\cudart64_*.dll C:\\Program Files\\nodejs\\cudart64_*.dll C:\\WINDOWS\\System32\\OpenSSH\\cudart64_*.dll C:\\Users\\user\\.cargo\\bin\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\cudart64_*.dll C:\\Program Files\\LLVM\\bin\\cudart64_*.dll C:\\Program Files\\GNU\\units\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\cudart64_*.dll C:\\Users\\user\\AppData\\Roaming\\npm\\cudart64_*.dll C:\\Windows\\System32\\OpenSSH\\cudart64_*.dll C:\\Program Files\\dotnet\\cudart64_*.dll C:\\Program Files\\Wolfram Research\\WolframScript\\cudart64_*.dll C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit\\cudart64_*.dll C:\\Program Files\\PowerShell\\7\\cudart64_*.dll C:\\Users\\user\\AppData\\Roaming\\Ruby32-x64\\bin\\cudart64_*.dll C:\\Users\\user\\.cargo\\bin\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Programs\\Microsoft VS Code\\bin\\cudart64_*.dll C:\\Program Files\\LLVM\\bin\\cudart64_*.dll C:\\Program Files\\GNU\\units\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\GitHubDesktop\\bin\\cudart64_*.dll C:\\Users\\user\\AppData\\Roaming\\npm\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll]"
time=2024-10-26T14:35:43.548+01:00 level=DEBUG source=gpu.go:562 msg="discovered GPU libraries" paths=[]
time=2024-10-26T14:35:43.575+01:00 level=DEBUG source=amd_hip_windows.go:88 msg=hipDriverGetVersion version=60241512
time=2024-10-26T14:35:43.575+01:00 level=DEBUG source=amd_common.go:18 msg="evaluating potential rocm lib dir C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama"
time=2024-10-26T14:35:43.575+01:00 level=DEBUG source=amd_common.go:61 msg="detected ROCM next to ollama executable C:\\Users\\user\\AppData\\Local\\Programs\\Ollama\\lib\\ollama"
time=2024-10-26T14:35:43.575+01:00 level=DEBUG source=amd_windows.go:73 msg="detected hip devices" count=2
time=2024-10-26T14:35:43.576+01:00 level=DEBUG source=amd_windows.go:93 msg="hip device" id=0 name="AMD Radeon(TM) Graphics" gfx=gfx1035
time=2024-10-26T14:35:44.091+01:00 level=WARN source=amd_windows.go:138 msg="amdgpu is not supported (supported types:[gfx1032])" gpu_type=gfx1035 gpu=0 library=C:\Users\user\AppData\Local\Programs\Ollama\lib\ollama
time=2024-10-26T14:35:44.091+01:00 level=DEBUG source=amd_windows.go:93 msg="hip device" id=1 name="AMD Radeon RX 6700S" gfx=gfx1032
time=2024-10-26T14:35:44.587+01:00 level=DEBUG source=amd_windows.go:146 msg="amdgpu is supported" gpu=1 gpu_type=gfx1032
time=2024-10-26T14:35:44.587+01:00 level=DEBUG source=amd_windows.go:149 msg="amdgpu memory" gpu=1 total="8.0 GiB"
time=2024-10-26T14:35:44.587+01:00 level=DEBUG source=amd_windows.go:150 msg="amdgpu memory" gpu=1 available="7.8 GiB"
time=2024-10-26T14:35:44.590+01:00 level=INFO source=types.go:123 msg="inference compute" id=1 library=rocm variant="" compute=gfx1032 driver=6.2 name="AMD Radeon RX 6700S" total="8.0 GiB" available="7.8 GiB"

So the GPU seems to be detected and supported. However, when I try to run ollama run llama3.2:1b I get:

image

Can you help?

OS

Windows

GPU

AMD

CPU

AMD

Ollama version

0.3.14-0-gc692dfd

likelovewant commented 2 hours ago

"AMD Radeon(TM) Graphics" gfx=gfx1035 "AMD Radeon RX 6700S" gfx=gfx1032 time=2024-10-26T14:35:44.587+01:00 level=DEBUG source=amd_windows.go:146 msg="amdgpu is supported" gpu=1 gpu_type=gfx1032

This happens due to the libs avialble at seperately withgfx1035 and gfx1032 . Dy set HIP_VISIBLE_DEVICE=1,0...(to defind the card using ).otherwise ,need to rebuild the rocmlibs for gfx1032 and gfx1035 together which will enable both dgpu and igup .ie gfx1032 and gfx1035 together . Also there is possible Navida drivers conflicts issue .searching for GPU discovery libraries for NVIDIA" time=2024-10-26T14:35:43.531+01:00 level=DEBUG source=gpu.go:505 msg="Searching for GPU library" name=nvml.dll

Astlaan commented 1 hour ago

Thank you. In the end I ended up closing because I had made a mistake, I had simply failed to copy the rocblas.dll file as well (only copies the TensileLibrary files).

It worked fine for me in the end. No need to copy rocblas.dll and the rocblas folder to the runners/roc....

likelovewant commented 1 hour ago

great .nice try.