Luax compile on M1 detects arch incorrectly for Zig.

[wmealing]

Tried doing an install on the M1 macs, which detect uname -m as an arm64 instead of the arch that zig uses download uses (aarch64).

Do you want me to submit a pull request with the fix ?

[wmealing]

After overcoming this, I found that ZIG doesn't compile any further on this architecture due to LTO issues.

11 fails with the error: (error: LTO is not yet supported with the Mach-O object format. More details: https://github.com/ziglang/zig/issues/8680)

12-dev fails with the error error: LTO requires using LLD

The offer for the PR is still open, I can't build luax at the moment due to the above issue.

I began looking at the build.lua however at the moment it is above my skills.

Thanks again.

[CDSoft]

I've no mac to test luax so it would be great if you can submit a pull request (I guess a quick fix in build_env.sh is enough).

LTO is disabled for mac binaries but not for temporary tools (such as lua). So this works on Linux to cross compile for mac but not on mac. I can disable LTO for these tools (no effect on the luax binaries, only the execution of the tests using a plain lua interpreter may take a few more milliseconds (no difference on my Linux PC)).

[CDSoft]

I've disabled LTO (for lua and lz4 only) on the dev branch. Can you rebase your fix on the dev branch?

[wmealing]

Tried it on 11 and 12, getting similiar errors.

info: zig can provide libc for related target aarch64-macos.11-none

`[2/673] CC ext/c/lz4/lib/lz4.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o.d ext/c/lz4/lib/lz4.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

[3/673] CC ext/c/lz4/lib/lz4hc.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o.d ext/c/lz4/lib/lz4hc.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4hc.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

[4/673] CC ext/c/lz4/lib/lz4file.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o.d ext/c/lz4/lib/lz4file.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4file.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

[5/673] CC ext/c/lz4/lib/lz4frame.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o.d ext/c/lz4/lib/lz4frame.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/lz4frame.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

[6/673] CC ext/c/lz4/lib/xxhash.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o.d ext/c/lz4/lib/xxhash.c -o .build/tmp/obj/lz4/ext/c/lz4/lib/xxhash.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

[7/673] CC ext/c/lz4/programs/bench.c FAILED: .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o . tools/build_env.sh; ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.12.0/zig-local-cache .zig/0.12.0/zig cc -target $ARCH-$OS-$LIBC -c -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax $LUA_CFLAGS -Werror -Wall -Wextra -Wno-constant-logical-operand -MD -MF .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o.d ext/c/lz4/programs/bench.c -o .build/tmp/obj/lz4/ext/c/lz4/programs/bench.o error: unable to find or provide libc for target 'aarch64-macos.11.7.1...14.1-gnu' info: zig can provide libc for related target aarch64-macos.11-none

` I'll try again (its late/early here) maybe i'll be able to understand a bit more when i'm more awake.

[wmealing]

Ok got past those errors, the LIBC has to be set to none in the tools/build_env.sh for mac os.. now a new error...

FAILED: .build/tmp/lua_runtime_bundle.dat . tools/build_env.sh; PATH=.build/tmp:$PATH LUA_PATH="./?.lua;.build/tmp/?.lua;libluax/?.lua;libluax/F/?.lua;libluax/L/?.lua;libluax/complex/?.lua;libluax/crypt/?.lua;libluax/fs/?.lua;libluax/imath/?.lua;libluax/import/?.lua;libluax/linenoise/?.lua;libluax/lpeg/?.lua;libluax/lz4/?.lua;libluax/mathx/?.lua;libluax/package/?.lua;libluax/ps/?.lua;libluax/qmath/?.lua;libluax/sh/?.lua;libluax/socket/?.lua;libluax/sys/?.lua;libluax/term/?.lua;libluax/version/?.lua" .build/tmp/lua -l tools/rc4_runtime luax/bundle.lua -lib -ascii .build/tmp/luax_config.lua libluax/F/F.lua libluax/L/L.lua libluax/complex/complex.lua libluax/crypt/crypt.lua libluax/fs/fs.lua libluax/imath/imath.lua libluax/import/import.lua libluax/linenoise/linenoise.lua libluax/lpeg/lpeg.lua libluax/lz4/lz4.lua libluax/mathx/mathx.lua libluax/package/package_hook.lua libluax/ps/ps.lua libluax/qmath/qmath.lua libluax/sh/sh.lua libluax/sys/sys.lua libluax/term/term.lua ext/c/lpeg/re.lua ext/c/luasocket/ftp.lua ext/c/luasocket/headers.lua ext/c/luasocket/http.lua ext/c/luasocket/ltn12.lua ext/c/luasocket/mbox.lua ext/c/luasocket/mime.lua ext/c/luasocket/smtp.lua ext/c/luasocket/socket.lua ext/c/luasocket/tp.lua ext/c/luasocket/url.lua ext/lua/argparse/argparse.lua ext/lua/cbor/cbor.lua ext/lua/inspect/inspect.lua ext/lua/json/json.lua ext/lua/serpent/serpent.lua > .build/tmp/lua_runtime_bundle.dat error: .build/tmp/luax_config.lua: File not found [339/675] CC[aarch64-linux-musl] ext/c/lz4/lib/lz4hc.c ninja: build stopped: subcommand failed.

However the file .build/tmp/luax_config.lua does exist and has contents.. working on that now.

[CDSoft]

The stat command may not work on mac as on linux. Could you please post the result of LC_ALL=C stat -L -c '%s;%Y;%X;%W;%F;%f' .build/tmp/luax_config.lua? or just LC_ALL=C stat -L .build/tmp/luax_config.lua?

[wmealing]

The OSX stat doesn't support -c, I had to use 'gstat' which is coreutils port renamed to gstat.

$ LC_ALL=C gstat -L -c '%s;%Y;%X;%W;%F;%f' .build/tmp/luax_config.lua 404;1704697291;1704697301;1704697291;regular file;81a4

$ LC_ALL=C stat -L .build/tmp/luax_config.lua 16777232 22569945 -rw-r--r-- 1 wmealing staff 0 404 "Jan 8 17:01:41 2024" "Jan 8 17:01:31 2024" "Jan 8 17:01:31 2024" "Jan 8 17:01:31 2024" 4096 8 0 .build/tmp/luax_config.lua

For completion, the contents are:

$ cat .build/tmp/luax_config.lua --@LIB local version = "2.25.2-2-geadbbdb" return { version = version, date = "2024-01-08", copyright = "LuaX "..version.." Copyright (C) 2021-2024 cdelord.fr/luax", authors = "Christophe Delord", magic_id = "LuaX", targets = {"x86_64-linux-musl", "x86_64-linux-gnu", "aarch64-linux-musl", "aarch64-linux-gnu", "x86_64-windows-gnu", "x86_64-macos-none", "aarch64-macos-none"}, }

[wmealing]

When i run the command by hand, and i know it exists: It still throws the error:

$ . tools/build_env.sh; PATH=.build/tmp:$PATH LUA_PATH="./?.lua;.build/tmp/?.lua;libluax/?.lua;libluax/F/?.lua;libluax/L/?.lua;libluax/complex/?.lua;libluax/crypt/?.lua;libluax/fs/?.lua;libluax/imath/?.lua;libluax/import/?.lua;libluax/linenoise/?.lua;libluax/lpeg/?.lua;libluax/lz4/?.lua;libluax/mathx/?.lua;libluax/package/?.lua;libluax/ps/?.lua;libluax/qmath/?.lua;libluax/sh/?.lua;libluax/socket/?.lua;libluax/sys/?.lua;libluax/term/?.lua;libluax/version/?.lua" .build/tmp/lua -l tools/rc4_runtime luax/bundle.lua -lib -ascii .build/tmp/luax_config.lua libluax/F/F.lua libluax/L/L.lua libluax/complex/complex.lua libluax/crypt/crypt.lua libluax/fs/fs.lua libluax/imath/imath.lua libluax/import/import.lua libluax/linenoise/linenoise.lua libluax/lpeg/lpeg.lua libluax/lz4/lz4.lua libluax/mathx/mathx.lua libluax/package/package_hook.lua libluax/ps/ps.lua libluax/qmath/qmath.lua libluax/sh/sh.lua libluax/sys/sys.lua libluax/term/term.lua ext/c/lpeg/re.lua ext/c/luasocket/ftp.lua ext/c/luasocket/headers.lua ext/c/luasocket/http.lua ext/c/luasocket/ltn12.lua ext/c/luasocket/mbox.lua ext/c/luasocket/mime.lua ext/c/luasocket/smtp.lua ext/c/luasocket/socket.lua ext/c/luasocket/tp.lua ext/c/luasocket/url.lua ext/lua/argparse/argparse.lua ext/lua/cbor/cbor.lua ext/lua/inspect/inspect.lua ext/lua/json/json.lua ext/lua/serpent/serpent.lua > .build/tmp/lua_runtime_bundle.dat

error: .build/tmp/luax_config.lua: File not found

This is the current tree : https://github.com/wmealing/luax/tree/dev

[CDSoft]

ok, so fs.stat shall use gstat on mac instead of stat.

Can you also confirm that uname -s returns Darwin and uname -m returns arm64? (build_env.sh and sys.lua must be updated)?

[wmealing]

bash-3.2$ uname -s Darwin

bash-3.2$ uname -m arm64

I updated build_env.sh but not sys.lua

[wmealing]

ok, so fs.stat shall use gstat on mac instead of stat.

I installed gstat via brew.. is that part of the problem ? stat that ships with OSX is the second command above

bash-3.2$ which stat /usr/bin/stat bash-3.2$ which gstat /opt/homebrew/bin//gstat

[CDSoft]

no, the problem is that fs.lua uses stat with the unsupported option -c. I've pushed a fix on the dev branch. Could you please test it?

[wmealing]

Needed an additional patch to tools/build_env.sh

See https://github.com/wmealing/luax/blob/dev/tools/build_env.sh#L39

The check for the LIBC. Otherwise the build fails with the comment in https://github.com/CDSoft/luax/issues/1#issuecomment-1880419705

Now with the patch and the build:

bash-3.2$ ninja [666/674] LD[aarch64-linux-musl] .build/tmp/run/luaxruntime-aarch64-linux-musl In file included from .zig/0.11.0/lib/libc/musl/crt/rcrt1.c:3: .zig/0.11.0/lib/libc/musl/crt/../ldso/dlstart.c:146:20: warning: a function declaration without a prototype is deprecated in all versions of C and is treated as a zero-parameter prototype in C2x, conflicting with a subsequent definition [-Wdeprecated-non-prototype] GETFUNCSYM(&dls2, __dls2, base+dyn[DT_PLTGOT]); ^ .zig/0.11.0/lib/libc/musl/crt/rcrt1.c:11:13: note: conflicting prototype is here hidden void __dls2(unsigned char base, size_t sp) ^ 1 warning generated. [670/674] LD[x86_64-windows-gnu] .build/tmp/run/luaxruntime-x86_64-windows-gnu.exe .zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:167:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] s->seen_eof = 1; ^ ~ .zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:176:22: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] else s->seen_eof = 1; ^ ~ .zig/0.11.0/lib/libc/mingw/stdio/mingw_wvfscanf.c:1628:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] ifp.is_string = 1; ^ ~ .zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:158:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] s->seen_eof = 1; ^ ~ .zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:167:22: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] else s->seen_eof = 1; ^ ~ .zig/0.11.0/lib/libc/mingw/stdio/mingw_vfscanf.c:1629:17: warning: implicit truncation from 'int' to a one-bit wide bit-field changes value from 1 to -1 [-Wsingle-bit-bitfield-constant-conversion] ifp.is_string = 1; ^ ~ 3 warnings generated. 3 warnings generated. [673/674] LUAX .build/bin/luax-pandoc FAILED: .build/bin/luax-pandoc .build/bin/luax -q -t pandoc -o .build/bin/luax-pandoc luax/luax.lua .build/bin/luax: No such file or directory No such file or directory [674/674] LUAX .build/bin/luax-lua FAILED: .build/bin/luax-lua .build/bin/luax -q -t lua -o .build/bin/luax-lua luax/luax.lua .build/bin/luax: No such file or directory No such file or directory ninja: build stopped: subcommand failed.

$ ls .build/bin/ luax luax-aarch64-macos-none luax-x86_64-macos-none luax-aarch64-linux-gnu luax-x86_64-linux-gnu luax-x86_64-windows-gnu.exe luax-aarch64-linux-musl luax-x86_64-linux-musl

./luax from here is :

$ file .build/bin/luax .build/bin/luax: Mach-O 64-bit executable arm64

However it crashes on execution.

bash-3.2$ ./.build/bin/luax ./.build/bin/luax: No such file or directory No such file or directory Segmentation fault: 11

Thanks for taking the time on this, i know how hard this kind of debugging is.

[CDSoft]

Do you have the same problem with the executable I've cross-compiled from Linux? https://cdelord.fr/hey/luax-aarch64-macos-none.tar.xz

[wmealing]

Same problem:

bash-3.2$ ./aarch64-macos-none/bin/luax ./aarch64-macos-none/bin/luax: No such file or directory No such file or directory Segmentation fault: 11

$ ./aarch64-macos-none/bin/luax-aarch64-macos-none ./aarch64-macos-none/bin/luax-aarch64-macos-none: No such file or directory No such file or directory Segmentation fault: 11

I had thought it might be some executable protection on OSX, but i have no evidence to back that up.

[wmealing]

This seems to work though:

bash-3.2$ ./aarch64-macos-none/bin/luax-lua print("hello") <-- typed this. hello <- echoed this.

[CDSoft]

If luax can not read itself, it won't work. luax-lua is a pure Lua implementation of some LuaX modules, run by lua, not luax.

So if luax can not read itself, its design does not work on mac (I had tested linux and windows only).

[wmealing]

Ah, that'll do it then, i need to go afk for a while. I will test any further suggestions you have overnight.

[CDSoft]

I'm afraid luax won't work on macos without a deep redesign.

[CDSoft]

To validate the hypothesis, can you run this test?

readmyself.c contains:

#include <stdio.h>

int main(int argc, char *argv[])
{
    printf("executable: %s\n", argv[0]);
    char buf[1024*1024];
    FILE *f = fopen(argv[0], "rb");
    size_t n = fread(&buf, 1, sizeof(buf), f);
    fclose(f);
    printf("size      : %zu\n", n);
}

Compiling and running this program prints this on Linux:

$ gcc readmyself.c -o readmyself && ./readmyself
executable: ./readmyself
size      : 16880

(should also work with clang)

Do you get a segmentation fault on macos?

[CDSoft]

I'm testing a simplified design for LuaX:

• only one binary for the current host (the default host is Linux, for other architectures the ninja file must be regenerated)
• programs "compiled" with luax require luax to be executed (thus the executable does not have to read itself)

This is in the luax3 branch.

e.g.:

$ git clone https://github.com/CDSoft/luax
$ cd luax
$ git checkout luax3
$ tools/bang    # regenerate build.ninja for your OS, requires lua 5.4
$ ninja
$ .build/bin/luax

[wmealing]

➜ ~ gcc readmyself.c -o readmyself && ./readmyself .executable: ./readmyself size : 33735

Oddly no crash.

➜ ~ clang readmyself.c -o readmyself && ./readmyself executable: ./readmyself size : 33687

No segmentation faults on either.

[wmealing]

I followed your above steps in the luax3 branch.

➜  luax git:(luax3) ✗ .build/bin/luax
 _               __  __  |  https://cdelord.fr/luax
| |   _   _  __ _\ \/ /  |
| |  | | | |/ _` |\  /   |  Version 2.25.2-3-gd9c7b6d (2024-01-11)
| |__| |_| | (_| |/  \   |  Powered by Lua 5.4
|_____\__,_|\__,_/_/\_\  |
                         |  Macos aarch64 none

[CDSoft]

Thanks for the test on macos!

[CDSoft]

The luax3 branch has been merged to master.

[wmealing]

Was able to build master.

It looks as though not all of the LTO fixes were pulled into master branch. There was something funky going on.

When I would try to build :

➜  luax git:(master) ninja                
[36/144] CC lua/lapi.c
FAILED: .build/tmp/obj/lua/lapi.o 
ZIG_GLOBAL_CACHE_DIR=$PWD/.zig/0.11.0/zig-global-cache ZIG_LOCAL_CACHE_DIR=$PWD/.zig/0.11.0/zig-local-cache .zig/0.11.0/zig cc -c -flto=thin -std=gnu2x -O3 -fPIC -I. -I.build/tmp -Ilua -Iext/c/lz4/lib -Ilibluax -Wno-constant-logical-operand -DLUA_USE_LINUX  -MD -MF .build/tmp/obj/lua/lapi.o.d lua/lapi.c -o .build/tmp/obj/lua/lapi.o
error: LTO is not yet supported with the Mach-O object format. More details: https://github.com/ziglang/zig/issues/8680
[45/144] CC ext/c/lz4/lib/lz4hc.c
ninja: build stopped: subcommand failed.

I traced (or think i did) this down to build.lua

diff --git a/build.lua b/build.lua
index 77cfd5a..b1138cd 100644
--- a/build.lua
+++ b/build.lua
@@ -237,7 +237,7 @@ local include_path = {
 local lto_opt = case(compiler) {
     zig   = "-flto=thin",
     gcc   = "-flto",
-    clang = "-flto=thin",
+    clang = "",
 }

Now when i run

➜  luax git:(master) ninja clean
[1/1] bang build.lua -o build.ninja
load build.lua
21 variables
35 rules
227 build statements
557 lines
43687 bytes
write build.ninja
[1/1] CLEAN .build
➜  luax git:(master) ✗ ninja
[144/144] CP .build/bin/luax

The luax binary is built and works.

➜  luax git:(master) ✗ .build/bin/luax       
 _               __  __  |  https://cdelord.fr/luax
| |   _   _  __ _\ \/ /  |
| |  | | | |/ _` |\  /   |  Version 3.0 (2024-01-11)
| |__| |_| | (_| |/  \   |  Powered by Lua 5.4
|_____\__,_|\__,_/_/\_\  |
                         |  Macos aarch64 none

>> 

[wmealing]

i realise the fix probably wont be in this location, because you likely want flto in clang on non osx systems.

[CDSoft]

It looks like you are running the ninja file for Linux (-DLUA_USE_LINUX is on the command line).

Did you run tools/bang -- macos-aarch64 before ninja? (I though I had updated the documentation but something has been lost, sorry)

[CDSoft]

tools/bang should be enough to detect the OS.

[CDSoft]

I'll close this issue. Feel free to reopen it if necessary.

The master branch should now have everything to compile LuaX on MacOS.