Problems in running the plugin

[JekSun97]

Please add SCons to build

[Adriankhl]

I would love to add a scons build, however, the llama.cpp dependency is a CMake/Make project and I believe it is quite hacky to integrate scons with CMake/Make.

It is possible to port a scons build for llama.cpp. It is a major effort though.

[JekSun97]

@Adriankhl I built it using CMake and Visual Studio, I now have the libraries godot-cpp.windows.debug.32.lib, libgodot_llm.windows..amd64.lib, llama.lib, ggml_static.lib, what should I do with all this next?

[Adriankhl]

@JekSun97 I am not familiar with visual studio myself, but it seems like you are getting close.

You need to find a way to build a dynamic library dll instead of static library lib, and if possible, set the cmake variable a CMAKE_BUILD_TYPE to Release (be aware that this is not used by MSBuild, you need to set the release build target somewhere else). Then, you should be able to build libgodot_llm.windows.release.amd64.dll (or rename the libgodot_llm... to this if you don't know how to set the cmake variable)

Finally, simply copy the dll and gdextension file to the addons folder of your godot project

├── addons
│   └── godot_llm
│       ├── bin
                        libgodot_llm.windows.release.amd64.dll
│       └── godot_llm.gdextension

[JekSun97]

It turns out that initially there was a compilation type for dll, now I decided to build libgodot_llm.windows..amd64 separately, but I get errors:

godot-cpp.windows.debug.32.lib(node.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(node.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(wrapped.obj) : error LNK2038: Inconsistency detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(wrapped.obj) : error LNK2038: Inconsistency detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(error_macros.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(error_macros.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(memory.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(memory.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(char_string.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(char_string.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(string.obj) : error LNK2038: Inconsistency detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(string.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(string_name.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(string_name.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(variant.obj) : error LNK2038: Inconsistency detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(variant.obj) : error LNK2038: Inconsistency detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(object.cpp.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(object.cpp.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(method_bind.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(method_bind.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(class_db.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(class_db.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(godot.obj) : error LNK2038: mismatch detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(godot.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_DynamicRelease" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(resource.obj) : error LNK2038: Inconsistency detected for "_ITERATOR_DEBUG_LEVEL": value "2" does not match value "0" in gdllama.obj 1>godot-cpp.windows.debug.32.lib(resource.obj) : error LNK2038: mismatch detected for "RuntimeLibrary": value "MDd_DynamicDebug" does not match value "MD_D"

[JekSun97]

I decided not to suffer and download the finished build, fortunately the license allows me to do this, once I test it I’ll post :)

[Adriankhl]

You are mixing debug build and release build, probably need to clean up project and rebuild everything under the same mode

[Adriankhl]

I decided not to suffer and download the finished build, fortunately the license allows me to do this, once I test it I’ll post :)

it is also already on the godot asset library, it is easiest to get it there

[JekSun97]

@Adriankhl I installed the plugin manually and via AssetLib, and I get the same error:

[Adriankhl]

@Adriankhl I installed the plugin manually and via AssetLib, and I get the same error:

May I confirm that you have created a new project, installed the plugin from the asset library, and this error pops up?

[Adriankhl]

If that is the case can you try to extract the dll file here and replace the addons/godot_llm/bin/libgodot_llm.windows.release.amd64.dll by this new file? libgodot_lib.zip

This is built without vulkan acceleration, just to see if the problem is related to gpu settings

[JekSun97]

May I confirm that you have created a new project, installed the plugin from the asset library, and this error pops up?

Yes

If that is the case can you try to extract the dll file here and replace the addons/godot_llm/bin/libgodot_llm.windows.release.amd64.dll by this new file? libgodot_lib.zip

This is built without vulkan acceleration, just to see if the problem is related to gpu settings

Same error :(

[Adriankhl]

:( Do you have Microsoft Visual C++ Redistributable installed?

[Adriankhl]

Let's try it one more time, here I also statically linked MSVC runtime. libgodot_llm.zip

[spiderbyte87]

Installing the addon crashes the editor for me.

[Adriankhl]

Installing the addon crashes the editor for me.

May I know if you have Microsoft Visual C++ Redistributable installed?

[Adriankhl]

I have created a new version of the plugin, this time I statically link the library to msvc runtime so no dependencies are needed, hopefully it fixes the problem.

[spiderbyte87]

Installing the addon crashes the editor for me.

May I know if you have Microsoft Visual C++ Redistributable installed?

I don't have MSVC. I tried the new version and it's still crashing as soon as the editor starts. I'm on Linux.. should I drop into Windows to see it the problem persists or would that not be it?

[Adriankhl]

Installing the addon crashes the editor for me.

May I know if you have Microsoft Visual C++ Redistributable installed?

I don't have MSVC. I tried the new version and it's still crashing as soon as the editor starts. I'm on Linux.. should I drop into Windows to see it the problem persists or would that not be it?

I see. It is strange that I am able to reproduce the problem on my linux laptop but it works fine under wsl on Windows, let me spend some time today to investigate the problem.

[Adriankhl]

Installing the addon crashes the editor for me.

May I know if you have Microsoft Visual C++ Redistributable installed?

I don't have MSVC. I tried the new version and it's still crashing as soon as the editor starts. I'm on Linux.. should I drop into Windows to see it the problem persists or would that not be it?

Could you download the new v0.2.1 zip file to see if this works for you. Really appreciate your feedback.

My test: this is super weird though, the plugin compiled on WSL simply doesn't work on native linux, I have tried WSL opensuse and ubuntu which have different glibc and gcc versions. The plugin compiled on native linux and virtual box Ubuntu works (even works for WSL linux)

I have checked the compile flags, the generated Makefiles are exactly the same on either platform. So probably something is wrong in the abi or generated symbol in the WSL environment, debugging this is probably beyond my current knowledge.

[JekSun97]

:( Do you have Microsoft Visual C++ Redistributable installed?

Yes

Let's try it one more time, here I also statically linked MSVC runtime. libgodot_llm.zip

[Adriankhl]

@JekSun97 @spiderbyte87 Finally I know what happened, there is a hidden default LLAMA_NATIVE option in the llama.cpp cmake build, which introduces optimization for my local cpu, but the binary becomes incompatible on many platforms. The Windows problem and Linux problem are the same issue, just that Windows handle the incompatibility slightly better so it doesn't crash (though it still doesn't provide useful debug information).

I have disabled the option and tested the new version on multiple PCs and VMs, I think now it should work.

[spiderbyte87]

@JekSun97 @spiderbyte87 Finally I know what happened, there is a hidden default LLAMA_NATIVE option in the llama.cpp cmake build, which introduces optimization for my local cpu, but the binary becomes incompatible on many platforms. The Windows problem and Linux problem are the same issue, just that Windows handle the incompatibility slightly better so it doesn't crash (though it still doesn't provide useful debug information).

I have disabled the option and tested the new version on multiple PCs and VMs, I think now it should work.

Will give it a try a little later today and let you know it if it works!

[spiderbyte87]

Yep it works without a hitch. I would highly recommend packaging this as a proper Godot addon with all the components in one place. Other than that, it's pretty great! I spent all yesterday getting Ollama working via POST, which works fine, but this is a far better solution! Thanks again!!

[JekSun97]

@Adriankhl can I test the dll?

[Adriankhl]

@Adriankhl can I test the dll?

Yes, you can grap it from the the zip file on the release page: https://github.com/Adriankhl/godot-llm/releases/tag/v0.2.2

You will find libgodot_llm.windows.release.amd64.dll in the bin folder

[JekSun97]

Now there is no error, but for some reason I couldn’t wait for an answer, I waited for about an hour, then closed the application.

I used this code:

extends Node

var lm = GDLlama.new()

func _ready() -> void:
if FileAccess.file_exists("res://Meta-Llama-3-8B-Instruct-Q5_K_M.gguf"):
lm.model_path="res://Meta-Llama-3-8B-Instruct-Q5_K_M.gguf"
var gen = lm.generate_text("5+5?")
print(gen)
else:
print("no file!")
pass

[Adriankhl]

Now there is no error, but for some reason I couldn’t wait for an answer, I waited for about an hour, then closed the application.

I used this code:

extends Node
> var lm = GDLlama.new()

func _ready() -> void:
if FileAccess.file_exists("res://Meta-Llama-3-8B-Instruct-Q5_K_M.gguf"):
lm.model_path="res://Meta-Llama-3-8B-Instruct-Q5_K_M.gguf"
var gen = lm.generate_text("5+5?")
print(gen)
else:
print("no file!")
pass

The default llama.cpp parameter lm.n_predict = -1 generates an infinite sequence, try setting lm.n_predict = 10 before running lm.generate_text

[JekSun97]

@Adriankhl Thank you! Now everything is working, I advise you to write the minimum verification code in the README for a quick start.

I wanted to know how you can make it so that the model can receive and respond in other languages? Maybe some special model is needed?

[Adriankhl]

@Adriankhl Thank you! Now everything is working, I advise you to write the minimum verification code in the README for a quick start.

Thanks for the suggestion! There is already a template godot project as a starting, but I think it is also good to have some simple coding example in the README.

I wanted to know how you can make it so that the model can receive and respond in other languages? Maybe some special model is needed?

It really depends on the model, llama 3 is already multillingual (depends on which language you want, it may or may not be good), and there are fine-tuned models which strengthen the output in a particular language. Bear in mind that the current stable version of this plugin doesn't convert utf8 correctly (fixed in the development version), so characters in other languages might be garbled.

[Adriankhl]

@JekSun97 I have fixed the text encoding bug, now at least llama 3 works in Chinese according to my testing.

And I have also updated the README documentation, I hope now it is clear enough 😺

[JekSun97]

@JekSun97 I have fixed the text encoding bug, now at least llama 3 works in Chinese according to my testing.

And I have also updated the README documentation, I hope now it is clear enough 😺

Best! I'll test it today

[Adriankhl]

Let's close this and use a new issue thread if there are further problems