Setting up a non-development runtime environment as a published .net application.

[AgentSmithers]

Description

Greetings everyone, I’ve done about a good six months of development diving into LlamaSharp including Microsoft's Kernel for rag. | validated my App works on two separate dev boxes using CUDA, CPU and Vulkan ATI but the challenge I am having is moving my application to a nondev machine via a published application. I got .Net core 8.0 runtime and SDK installed and upon running my binary | get an error stating that I don’t have a CPU backend installed... reading around the LlamaCPP and LlamaSharp(here) GitHub issues I didn't spot anything that stuck out when it came to the deployment of applications outside of a dev environment but I did see a few post recommending OpenBlas or CLBLAS as a CPU backend that involved recompiling LlamaCPP which I am not sure if that’s actually required for a LlamaSharp deployment but I can’t say for certain because on my dev box ljust run install-Package Llama.backend.cpu and everything works from there and I’m not sure if that downloads and configures OpenBlas behind the scenes. Can anyone confirm if I am going down the correct road or if I should be looking into something else? Any referenced links are also appreciated. I think my main struggle is finding detailed documentation on setting up a runtime environment vs. a dev environment. Thank you everyone in advanced!

[martindevans]

There have been issue with publishing final apps before - which haven't been fixed yet. They seem to be related to the location of the binaries being different in the development app and the final built app.

Perhaps try adding a logger to the loading system (NativeLibraryConfig.All.WithLogger) and printing that out in your built application. It should show the exact paths of the DLLs it is trying to load, and you can cross check those with where the files actually are on disk.

[AgentSmithers]

Thank you, this is extremely helpful to get some feedback. I thought I was doing something wrong. So theres no need to download OpenBlas or CL then to get this thing to fly, correct? It's all in ensuring the DLL's are deployed correctly and located in the right folder? Is there an action item where I need to take a qualifying LLama.dll and rename it to LibLlama.dll? and if so does that need to be placed in the same directory as the binary or one of its sub directories? Thanks again for your feedback, I have a route to explore now.

[martindevans]

In theory this should all just work :tm:, so I can't give you a definite answer to what you'll need to do exactly. I'm fairly sure you won't need to rename it to libllama.dll at least, that's a Linux naming convention that's not needed for Windows DLLs.

Once you've done the logging thing if you post your results here I should be able to tell you exactly what to copy where, and after that maybe we can work out why the files aren't being deployed to the correct locations automatically :)

[AgentSmithers]

Runtime box is on 2019 Server CORE, not gui.

I added this to the start of my project, have confirmed x64 vc2015 runtime has been installed. NativeLibraryConfig.All.WithLogCallback((level, message) => { Console.Write($"[llama {level}]: {message}"); });

Here's a screenshot of what I have so far, on the left is my working Dev box... and on the right, the Runtime box running through RDP. The DLL reflects that the path is located but does not give any feedback as to why llama.dll fails to load. I'm going to continue experimenting, maybe dumpbin will reflect the underlaying reason why.

update: No dice, dumpbin reflected VC140*.dll's which is why I figured maybe the runtime was missing for one of the dependencies. Still receiving the same error.

[martindevans]

Ah I initially assumed this was an issue with the DLLs simply being in the wrong place, but that seems to be totally wrong! Failed to load means NativeLibrary.TryLoad was called (a runtime method) but the runtime failed to load the DLL.

Dumpbin would have been my next suggestion, but you beat me to it! There was nothing else missing besides the vc140 runtime?

[AgentSmithers]

Correct, not as far as I could tell, I ran dumpbin /dependents and spotted that VC140's DLLs were listed but after installing VC++ 2015 Runtime both for x64 and x86, same error. Dumpbin didn't say anything explicit about anything missing from the system, is there a preferred command to run to confirm this that I may not be aware of? I'm open to any other suggestions.

[martindevans]

I'm a bit stumped by that I'm afraid. The only other thing I can think of is AVX support, but we distribute a DLL with no AVX requirements at all to account for that and it failed to load as well!

[AgentSmithers]

Any idea on what method can be used to find the underlaying loading issues? I trued fusion but it did not display anything in the logs and am not sure if It should reflect their or if I need to try some other method to get to the root cause of the DLL failing.

[martindevans]

A few ideas:

• You could try dependency walker
• Sometimes when you download a file windows marks it as blocked, and won't allow it to be opened. Maybe check that for the DLLs?
• Check the file hash to make sure it's not been corrupted during download.
• You could try cloning llama.cpp and compiling the binaries on the target machine (make sure you use exactly the correct version, there is no version-to-version compatibility).