search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
1.89k stars 416 forks source link

Building llama_cpp container does not produce the compiled C++ version of llama.cpp executables #509

Open amasciotta opened 1 month ago

amasciotta commented 1 month ago

Hello! I am using the prebuilt container dustynv/llama_cpp, which contains the built C++ executables inside /opt/llama.cpp.

However, trying to rebuild it from scratch to get some fixes from the main repo, I obtain a container that has no /opt/llama.cpp folder.

I suspect that #422 broke something...

dusty-nv commented 1 month ago

Hi @amasciotta, yes it builds llama_cpp_python from https://github.com/abetlen/llama-cpp-python which has llama.cpp as submodule. For now if you need to retain the C++ stuff, you just may want to build it, it is straightforward to compile these days with -DLLAMA_CUBLAS=on -DLLAMA_CUDA_F16=1

amasciotta commented 1 month ago

I will definitely try to build it myself. Any other suggestion on how to set compile options to work best on Jetson Xavier and Orin?

Finally, a note: LLAMA_CUBLAS is deprecated, in favour of LLAMA_CUDA.