dschilling
commented
4 days ago I am experiencing the same issue on Ubuntu. When running the example application, even though the CUDA native library is present, it never checks to see if it exists, instead only looking for the Vulcan library.
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.1 LTS
Release: 24.04
Codename: noble
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ dotnet --version
8.0.110
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-bfdfe6b5-e9c2-ff48-bf26-22e2f02b9757)
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ git status
HEAD detached at v0.19.0
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: Program.cs
no changes added to commit (use "git add" and/or "git commit -a")
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ git diff
diff --git a/LLama.Examples/Program.cs b/LLama.Examples/Program.cs
index 63114120..58ebeb6e 100644
--- a/LLama.Examples/Program.cs
+++ b/LLama.Examples/Program.cs
@@ -18,7 +18,7 @@ AnsiConsole.MarkupLineInterpolated(
""");
// Configure logging. Change this to `true` to see log messages from llama.cpp
-var showLLamaCppLogs = false;
+var showLLamaCppLogs = true;
NativeLibraryConfig
.All
.WithLogCallback((level, message) =>
@@ -37,4 +37,4 @@ NativeLibraryConfig
// Calling this method forces loading to occur now.
NativeApi.llama_empty_call();
-await ExampleRunner.Run();
\ No newline at end of file
+//await ExampleRunner.Run();
\ No newline at end of file
daniel@reteep:~/code/LLamaSharp/LLama.Examples$ dotnet run --framework net8.0
/home/daniel/code/LLamaSharp/LLama.Examples/Examples/QuantizeModel.cs(5,34): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread. [/home/daniel/code/LLamaSharp/LLama.Examples/LLama.Examples.csproj::TargetFramework=net8.0]
/home/daniel/code/LLamaSharp/LLama.Examples/Examples/SemanticKernelMemory.cs(18,17): warning CS0219: The variable 'seed' is assigned but its value is never used [/home/daniel/code/LLamaSharp/LLama.Examples/LLama.Examples.csproj::TargetFramework=net8.0]
/home/daniel/code/LLamaSharp/LLama.Examples/Examples/SemanticKernelHomeAutomation.cs(126,21): warning CS8602: Dereference of a possibly null reference. [/home/daniel/code/LLamaSharp/LLama.Examples/LLama.Examples.csproj::TargetFramework=net8.0]
/home/daniel/code/LLamaSharp/LLama.Examples/Examples/LlavaInteractiveModeExecute.cs(96,50): warning CS8602: Dereference of a possibly null reference. [/home/daniel/code/LLamaSharp/LLama.Examples/LLama.Examples.csproj::TargetFramework=net8.0]
======================================================================================================
__ __ ____ __
/\ \ /\ \ /\ _`\ /\ \
\ \ \ \ \ \ __ ___ ___ __ \ \,\L\_\\ \ \___ __ _ __ _____
\ \ \ __\ \ \ __ /'__`\ /' __` __`\ /'__`\ \/_\__ \ \ \ _ `\ /'__`\ /\` __\/\ __`\
\ \ \L\ \\ \ \L\ \/\ \L\.\_ /\ \/\ \/\ \ /\ \L\.\_ /\ \L\ \\ \ \ \ \ /\ \L\.\_\ \ \/ \ \ \L\ \
\ \____/ \ \____/\ \__/.\_\\ \_\ \_\ \_\\ \__/.\_\\ `\____\\ \_\ \_\\ \__/.\_\\ \_\ \ \ ,__/
\/___/ \/___/ \/__/\/_/ \/_/\/_/\/_/ \/__/\/_/ \/_____/ \/_/\/_/ \/__/\/_/ \/_/ \ \ \/
========================================================================================= \ \_\ ======
\/_/
[llama Debug]: Beginning dry run for llama...
[llama Debug]: Loading library: 'llama'
[llama Info]: Detected OS Platform: 'LINUX'
[llama Debug]: Detected OS string: 'linux-x64'
[llama Debug]: Detected extension string: '.so'
[llama Debug]: Detected prefix string: 'lib'
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Debug]: Got relative library path 'runtimes/linux-x64/native/vulkan/libllama.so' from local with (NativeLibraryName: LLama, UseCuda: False, UseVulkan: True, AvxLevel: None), trying to load it...
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so' for relative path 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded dependency 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllama.so' for relative path 'runtimes/linux-x64/native/vulkan/libllama.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllama.so'
[llama Debug]: Beginning dry run for llava_shared...
[llama Debug]: Loading library: 'llava_shared'
[llama Info]: Detected OS Platform: 'LINUX'
[llama Debug]: Detected OS string: 'linux-x64'
[llama Debug]: Detected extension string: '.so'
[llama Debug]: Detected prefix string: 'lib'
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLava
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLava
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Debug]: Got relative library path 'runtimes/linux-x64/native/vulkan/libllava_shared.so' from local with (NativeLibraryName: LLava, UseCuda: False, UseVulkan: True, AvxLevel: None), trying to load it...
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so' for relative path 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded dependency 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllava_shared.so' for relative path 'runtimes/linux-x64/native/vulkan/libllava_shared.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllava_shared.so'
[llama Debug]: Loading library: 'llama'
[llama Info]: Detected OS Platform: 'LINUX'
[llama Debug]: Detected OS string: 'linux-x64'
[llama Debug]: Detected extension string: '.so'
[llama Debug]: Detected prefix string: 'lib'
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Info]: NativeLibraryConfig Description:
- LibraryName: LLama
- Path: ''
- PreferCuda: True
- PreferVulkan: True
- PreferredAvxLevel: AVX2
- AllowFallback: True
- SkipCheck: False
- SearchDirectories and Priorities: { ./ }
[llama Debug]: Got relative library path 'runtimes/linux-x64/native/vulkan/libllama.so' from local with (NativeLibraryName: LLama, UseCuda: False, UseVulkan: True, AvxLevel: None), trying to load it...
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so' for relative path 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libggml.so'
[llama Info]: Successfully loaded dependency 'runtimes/linux-x64/native/vulkan/libggml.so'
[llama Debug]: Found full path file '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllama.so' for relative path 'runtimes/linux-x64/native/vulkan/libllama.so'
[llama Info]: Successfully loaded '/home/daniel/code/LLamaSharp/LLama.Examples/bin/Debug/net8.0/runtimes/linux-x64/native/vulkan/libllama.so'
Description
on my laptop with a gtx 1060 in it, i have the cpu backend and the cuda backend installed. However the native all log does not show it even loading cuda, but instead shows it trying to load vulkan (not installed) and then eventually defaulting to avx2 and using cpu only. It also allows me to offload gpu layers, but never goes into the gpu buffer.
Reproduction Steps
https://pastebin.com/TmsUg95V this is the pastebin log
Environment & Configuration
GTX 1060 6GB
Known Workarounds
No response