Support linux (and consider docker)

[cztomsik]

which implies having headless linux build

[Nindaleth]

Compilation of the headless build (from feat/headless) is currently unsuccessful:

Fetch Packages [1/1] ava-sqlite... /home/user/src/ava/build.zig.zon:8:21: error: hash mismatch: expected: 122013caee042269ef748bd0631d9c92afdf856755743e8189ac8ad758b65caa8253, found: 12201c1662ca2f679d1e350f8010bef3ef7c5f0f50b739a83b743c3fd7b034a67921
            .hash = "122013caee042269ef748bd0631d9c92afdf856755743e8189ac8ad758b65caa8253",
                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

That's from your own other repo, so I just took the liberty of changing expected to current. Afterwards there are the following issues:

$ zig build -Dheadless=true
/home/user/src/ava/build.zig:22:27: error: unable to load '/home/user/src/ava/src/macos/BuildMacOS.zig': FileNotFound
        .macos => @import("src/macos/BuildMacOS.zig").create(b),
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/src/ava/build.zig:81:14: error: member function expected 2 argument(s), found 1
        llama.addCSourceFiles(.{ .files = &.{"llama.cpp/ggml-metal.m"}, .flags = cflags.items });
        ~~~~~^~~~~~~~~~~~~~~~
/home/user/bin/lib/std/Build/Step/Compile.zig:891:5: note: function declared here
pub fn addCSourceFiles(self: *Compile, files: []const []const u8, flags: []const []const u8) void {
~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
referenced by:
    build: /home/user/src/ava/build.zig:17:9
    runBuild__anon_7181: /home/user/bin/lib/std/Build.zig:1639:37
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

The first one is a simple fix of git mv src/macos/BuildMac{os,OS}.zig. I don't have the knowledge to fix the others.

[cztomsik]

I think you've forgot to git checkout feat/headless because the code-path you're referring to is commented out in that branch.

[Nindaleth]

You're right, that's on me, sorry for the confusion. After the branch switch, the ava-sqlite checksum is still not matching. After updating the checksum, I get the following:

/home/user/src/ava/build.zig:3:26: error: root struct of file 'Build' has no member named 'ResolvedTarget'
pub var target: std.Build.ResolvedTarget = undefined;
                ~~~~~~~~~^~~~~~~~~~~~~~~
referenced by:
    build: /home/user/src/ava/build.zig:14:5
    runBuild__anon_7179: /home/user/bin/lib/std/Build.zig:1639:37
    remaining reference traces hidden; use '-freference-trace' to see all reference traces

I'm using the Zig official release 0.11.0 and placed both the binary and the lib into my ~/bin/. Should I use the current master instead of the latest release?

[cztomsik]

You need latest (nightly) build of Zig. Sorry, there are too many important things in nightly. (But I am not sure about the hash, I don't get any errors but maybe it's because nightly too?)

[Nindaleth]

OK, here's a try with the nightly build of Zig (2024-01-10):

install
└─ install llama
   └─ zig build-lib llama ReleaseFast native 20 errors
/home/user/src/ava/llama.cpp/ggml.c:347:5: error: call to undeclared function 'clock_gettime'; ISO C99 and later do not support implicit function declarations
    clock_gettime(CLOCK_MONOTONIC, &ts);
    ^
/home/user/src/ava/llama.cpp/ggml.c:347:19: error: use of undeclared identifier 'CLOCK_MONOTONIC'
    clock_gettime(CLOCK_MONOTONIC, &ts);
                  ^
/home/user/src/ava/llama.cpp/ggml.c:353:5: error: call to undeclared function 'clock_gettime'; ISO C99 and later do not support implicit function declarations
    clock_gettime(CLOCK_MONOTONIC, &ts);
    ^
/home/user/src/ava/llama.cpp/ggml.c:353:19: error: use of undeclared identifier 'CLOCK_MONOTONIC'
    clock_gettime(CLOCK_MONOTONIC, &ts);
                  ^
/home/user/src/ava/llama.cpp/ggml.c:11583:59: error: use of undeclared identifier 'M_PI'
    return n_dims * logf(n_orig_ctx / (n_rot * 2 * (float)M_PI)) / (2 * logf(base));
                                                          ^
/home/user/src/ava/llama.cpp/ggml.c:16163:22: error: call to undeclared function 'CPU_ALLOC_SIZE'; ISO C99 and later do not support implicit function declarations
    size_t setsize = CPU_ALLOC_SIZE(g_state.numa.total_cpus);
                     ^
/home/user/src/ava/llama.cpp/ggml.c:16165:24: error: call to undeclared function 'CPU_ALLOC'; ISO C99 and later do not support implicit function declarations
    cpu_set_t * cpus = CPU_ALLOC(g_state.numa.total_cpus);
                       ^
/home/user/src/ava/llama.cpp/ggml.c:16165:17: error: incompatible integer to pointer conversion initializing 'cpu_set_t *' with an expression of type 'int'
    cpu_set_t * cpus = CPU_ALLOC(g_state.numa.total_cpus);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/src/ava/llama.cpp/ggml.c:16166:5: error: call to undeclared function 'CPU_ZERO_S'; ISO C99 and later do not support implicit function declarations
    CPU_ZERO_S(setsize, cpus);
    ^
/home/user/src/ava/llama.cpp/ggml.c:16168:9: error: call to undeclared function 'CPU_SET_S'; ISO C99 and later do not support implicit function declarations
        CPU_SET_S(node->cpus[i], setsize, cpus);
        ^
/home/user/src/ava/llama.cpp/ggml.c:16171:14: error: call to undeclared function 'pthread_setaffinity_np'; ISO C99 and later do not support implicit function declarations
    int rv = pthread_setaffinity_np(pthread_self(), setsize, cpus);
             ^
/home/user/src/ava/llama.cpp/ggml.c:16177:5: error: call to undeclared function 'CPU_FREE'; ISO C99 and later do not support implicit function declarations
    CPU_FREE(cpus);
    ^
/home/user/src/ava/llama.cpp/ggml.c:16185:22: error: call to undeclared function 'CPU_ALLOC_SIZE'; ISO C99 and later do not support implicit function declarations
    size_t setsize = CPU_ALLOC_SIZE(g_state.numa.total_cpus);
                     ^
/home/user/src/ava/llama.cpp/ggml.c:16187:24: error: call to undeclared function 'CPU_ALLOC'; ISO C99 and later do not support implicit function declarations
    cpu_set_t * cpus = CPU_ALLOC(g_state.numa.total_cpus);
                       ^
/home/user/src/ava/llama.cpp/ggml.c:16187:17: error: incompatible integer to pointer conversion initializing 'cpu_set_t *' with an expression of type 'int'
    cpu_set_t * cpus = CPU_ALLOC(g_state.numa.total_cpus);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user/src/ava/llama.cpp/ggml.c:16188:5: error: call to undeclared function 'CPU_ZERO_S'; ISO C99 and later do not support implicit function declarations
    CPU_ZERO_S(setsize, cpus);
    ^
/home/user/src/ava/llama.cpp/ggml.c:16190:9: error: call to undeclared function 'CPU_SET_S'; ISO C99 and later do not support implicit function declarations
        CPU_SET_S(i, setsize, cpus);
        ^
/home/user/src/ava/llama.cpp/ggml.c:16193:14: error: call to undeclared function 'pthread_setaffinity_np'; ISO C99 and later do not support implicit function declarations
    int rv = pthread_setaffinity_np(pthread_self(), setsize, cpus);
             ^
/home/user/src/ava/llama.cpp/ggml.c:16199:5: error: call to undeclared function 'CPU_FREE'; ISO C99 and later do not support implicit function declarations
    CPU_FREE(cpus);
    ^
:1:1: error: too many errors emitted, stopping now
error: the following command failed with 20 compilation errors:
/home/user/bin/zig build-lib -cflags -std=c11 -Ofast -DNDEBUG -DGGML_USE_K_QUANTS -fPIC -- /home/user/src/ava/llama.cpp/ggml.c /home/user/src/ava/llama.cpp/ggml-alloc.c /home/user/src/ava/llama.cpp/ggml-backend.c /home/user/src/ava/llama.cpp/ggml-quants.c -cflags -std=c++11 -Ofast -DNDEBUG -fPIC -- /home/user/src/ava/llama.cpp/llama.cpp -OReleaseFast -I /home/user/src/ava/llama.cpp -lc++ --cache-dir /home/user/src/ava/zig-cache --global-cache-dir /home/user/.cache/zig --name llama -static --listen=- 
Build Summary: 0/5 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
├─ install llama transitive failure
│  └─ zig build-lib llama ReleaseFast native 20 errors
└─ install ava transitive failure
   └─ zig build-exe ava Debug native transitive failure
      └─ zig build-lib llama ReleaseFast native (reused)
error: the following build command failed with exit code 1:
/home/user/src/ava/zig-cache/o/f0de03c6e5ff1e138f0c4a1e0c919340/build /home/user/bin/zig /home/user/src/ava /home/user/src/ava/zig-cache /home/user/.cache/zig --seed 0xd163ef71 -Dheadless=true

[cztomsik]

I think I can replicate this, looking into it right now :)

[Nindaleth]

After updating zig to 0.12.0-dev.2296+46d592e48 and feat/headless branch to 11acb05, I can build and run Ava successfully! :rocket:

$ zig build -Dheadless=true
...

$ ./bin/ava 
18:49:23 debug default: Starting the server
(...)

  /\ \  / /\             Server running
 /--\ \/ /--\            http://127.0.0.1:3002
 _____________________________________________

$ curl -X POST localhost:3002/api/models -d '{"id": 1, "name": "Mistral-7B-Instruct-v0.2", "path": "..."}'
...

$ curl -X POST localhost:3002/api/generate -d '{"model_id": 1, "prompt": Free libre open source software is great because", "sampling": {"top_k": 40, "top_p": 0.8, "temperature": 0.7, "repeat_n_last": 256, "repeat_penalty": 1.1, "stop_eos": true}}'
{"status":"Waiting for the model..."}
{"status":"Reading the history..."}
{"status":"Reading the history... (0/9)"}
{"status":""}
{"content":" it"}
{"content":" enables"}
{"content":" everyone"}
...

[cztomsik]

perfect :) I think the branch should now compile macos & windows again (both GUI & headless)

regarding the linux, it's just a bit unclear to me why the binary is 34M big... I'm not sure if it's just because I'm cross-compiling or where is the problem. Both macos & windows builds are similar size before this change.

[cztomsik]

I had some troubles with GUI, but headless builds now using github actions for headless linux/windows builds) https://github.com/cztomsik/ava/actions/runs/7597511311

(there's some problem with macos, which I haven't seen before so that will take some time)

[Nindaleth]

regarding the linux, it's just a bit unclear to me why the binary is 34M big

I get similar size too when direct-compiling. On the linked Linux artifact, even with -Doptimize=ReleaseFast applied, I can see the following:

$ file ava_x86_64
ava_x86_64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

Should ReleaseFast contain the debug info too? After stripping, the binary gets from 30 MB to 4.5 MB, which looks suspiciously similar to the Windows binary size, so maybe it's just caused by that debug info.

[cztomsik]

Ok, I think this should be fixed (already merged) And thanks for the hint :)

https://github.com/cztomsik/ava/actions/runs/7600130509

[cztomsik]

I've migrated the website to github pages, and added a link to the linux build. This is still not done, but it's a step.

[cztomsik]

Closing, linux should work and given that it's single binary it should be easy for anyone to create their own docker image. I am totally fine if someone wants to maintain their images on dockerhub or elsewhere but it's not something I have time to do myself.