[BUG]: Linux cuda version detection could be incorrect

[AsakusaRinne]

Description

Sometimes the cuda version detection in linux is not correct.

Reproduction Steps

1. pull and run the docker nvidia/cuda:11.7.1-devel-ubuntu22.04.
2. Run the LLamaSharp examples

Environment & Configuration

• Operating system: Ubuntu
• .NET runtime version: 8.0
• LLamaSharp version: master
• CUDA version (if you are using cuda backend): 11.7
• CPU & GPU device: 2080Ti

Known Workarounds

SkipCheck or WithLibrary is a quick fix but not good.

The main reason is that the version.json or version.txt does not exist. A possible solution is to create a repo about calling nvml APIs but that will cost a lot of time.

[Hyp3rSoniX]

Not sure if this is an okay solution for you but for my usage, I just added an additional environment-var check for the cuda major version: https://gitlab.um-a.one/ai/LLamaSharp/-/blob/3baffe56e49bb823b01c9dff9c463a2c21ab000e/LLama/Native/NativeApi.Load.cs#L121

            else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
            {
                var manualVersionSet = Environment.GetEnvironmentVariable("CUDA_MAJOR_VERSION");
                if(!string.IsNullOrEmpty(manualVersionSet)){
                    return int.Parse(manualVersionSet);
                }
                // Try the default first
                cudaPath = "/usr/local/bin/cuda";
                version = GetCudaVersionFromPath(cudaPath);
                if (string.IsNullOrEmpty(version))
                {
                    cudaPath = Environment.GetEnvironmentVariable("LD_LIBRARY_PATH");
                    if (cudaPath is null)
                    {
                        return -1;
                    }
                    foreach (var path in cudaPath.Split(':'))
                    {
                        version = GetCudaVersionFromPath(Path.Combine(path, ".."));
                        if (string.IsNullOrEmpty(version))
                        {
                            break;
                        }
                    }
                }
            }

I just set the same environment variable in the dockerfile, and the check always works. One can also set it via the docker run -e command, or export it in the shell.

I don't think you need to implement something super complex just to parse some nvidia tool result. You can expect an environment variable or maybe even a config set within the project, that simply tells you what cuda version is installed. Mention it in the docu and you're done.

Just my two cents.

[AsakusaRinne]

Yeah makes sense, that's a good solution. Thank you a lot for your suggestions!