An Unreal focused API wrapper for llama.cpp to support embedding LLMs into your games locally. Forked from upstream to focus on improved API with wider support for builds (CPU, CUDA, Android, Mac).
Early releases, api still pretty unstable YMMV.
NB: currently has #7 issue which may require you to do your own static llama.cpp build until resolved.
Llama-Unreal-UEx.x-vx.x.x.7z
link which contains compiled binaries, not the Source Code (zip) link.ggml.dll
and llama.dll
to your project binaries folder for it to run correctly. (v0.5.0 issue)If you want to do builds for your own use case or replace the llama.cpp backend. Note that these build instructions should be run from the cloned llama.cpp root directory, not the plugin root.
Forked Plugin Llama.cpp was built from git hash: 2f3c1466ff46a2413b0e363a5005c46538186ee6
With the following build commands for windows (cpu build only, CUDA ignored, see upstream for GPU version):
mkdir build
cd build/
cmake ..
cmake --build . --config Release -j --verbose
ATM built for CUDA 12.2 runtime
cuda
branch if you want cuda enabled.cudart.lib
cublas.lib
and cuda.lib
into your libraries/win64 path. These are ignored atm.bTryToUseCuda = true;
is set in LlamaCore.build.cs to add CUDA libs to build.-DBUILD_SHARED_LIBS=ON
mkdir build
cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j --verbose
mkdir build
cd build/
cmake .. -DBUILD_SHARED_LIBS=ON
cmake --build . --config Release -j --verbose
For Android build see: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#android
mkdir build-android
cd build-android
export NDK=<your_ndk_directory>
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod ..
$ make
Then the .so or .lib file was copied into e.g. ThirdParty/LlamaCpp/Win64/cpu
directory and all the .h files were copied to the Includes
directory.