Help installing any version on Jetson AGX Xavier

[ark626]

LocalAI version:

v1.25.0 Environment, CPU architecture, OS, and Version: CPU aarch64, Ubuntu 20.04

Describe the bug

While trying to run BUILD_GRPC_FOR_BACKEND_LLAMA=ON make build after adding the following fix to the tagged version v1.25.0

git diff ggml.h:

diff --git a/ggml.h b/ggml.h
index 255541d..368ac2e 100644
--- a/ggml.h
+++ b/ggml.h
@@ -212,9 +212,11 @@
 extern "C" {
 #endif

-#ifdef __ARM_NEON
+#if defined(_ARM_NEON) && !defined(CUDACC_)
     // we use the built-in 16-bit float type
     typedef __fp16 ggml_fp16_t;
+#elif defined(_ARM_NEON) && defined(CUDACC_)
+    typedef half ggml_fp16_t;
 #else
     typedef uint16_t ggml_fp16_t;
 #endif

I get the following followup error:

cd build && cp -rf CMakeFiles/ggml.dir/k_quants.c.o ../ggllm.cpp/k_quants.o
ar src libggllm.a ggllm.cpp/libfalcon.o ggllm.cpp/cmpnct_unicode.o ggllm.cpp/ggml.o ggllm.cpp/k_quants.o  ggllm.cpp/falcon_common.o falcon_binding.o
make[1]: Leaving directory '/mnt/external/LocalAI/go-ggllm'
CGO_LDFLAGS="" C_INCLUDE_PATH=/mnt/external/LocalAI/go-ggllm LIBRARY_PATH=/mnt/external/LocalAI/go-ggllm \
go build -ldflags " -X "github.com/go-skynet/LocalAI/internal.Version=v1.25.0" -X "github.com/go-skynet/LocalAI/internal.Commit=9e5fb2996582bc45e5a9cbe6f8668e7f1557c15a"" -tags "" -o backend-assets/grpc/falcon ./cmd/grpc/falcon/
# github.com/go-skynet/LocalAI/cmd/grpc/falcon
/usr/lib/go-1.21/pkg/tool/linux_arm64/link: running g++ failed: exit status 1
/usr/bin/ld: /mnt/external/LocalAI/go-ggllm/libggllm.a(libfalcon.o): in function `falcon_prepare_buffers':
libfalcon.cpp:(.text+0x53a4): undefined reference to `ggml_cuda_host_free'

Additional context Is there any version capable of getting compiled on Jetson Xavier AGX?

[mudler]

I think libfalcon is not compatible with CUDA.

As a workaround you can actually skip building the falcon backend:

make GRPC_BACKENDS=backend-assets/grpc/llama-cpp build

see also the docs for example: https://localai.io/basics/build/#build-only-a-single-backend