ksh @2e849e6 and later do not build on macOS 13.6.6

[posguy99]

As above. I ran bin/package clean before the build.

Error is:

# ... making src/lib/libsum ...
+ mamake -C lib/libsum install
dyld[6820]: Library not loaded: libast.6.dylib
  Referenced from: <DF884FB7-2958-3808-AFE7-3EB00A977DD8> /Users/mwilson/src/ksh-test/arch/darwin.arm64-64/dyn/bin/mamake
  Reason: tried: 'libast.6.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibast.6.dylib' (no such file), 'libast.6.dylib' (no such file), '/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file), '/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file)
mamake: *** exit code 134 making lib/libsum
mamake: *** exit code 1 making all
package: make failed at Sat Mar 30 11:16:49 PDT 2024 in /Users/mwilson/src/ksh-test/arch/darwin.arm64-64

The make.out file is attached.

make.out.txt

[posguy99]

@f2bc1f4 does build without errors.

[posguy99]

And it DID build libast and it's sitting where it should be.

$ ll
total 1480
-rwxr-xr-x  1 mwilson  staff  754144 Mar 30 11:16 libast.6.0.dylib*
lrwxr-xr-x  1 mwilson  staff      16 Mar 30 11:16 libast.6.dylib@ -> libast.6.0.dylib
lrwxr-xr-x  1 mwilson  staff      16 Mar 30 11:16 libast.dylib@ -> libast.6.0.dylib

[posguy99]

More reading of the log... the invocation of mamake is what's actually failing, isn't it?

# src/lib/libast/Mamfile: 5075-5080: make ${INSTALLROOT}/dyn/bin/mamake
+ LDFLAGS='  '
+ export LDFLAGS
+ cat /Users/mwilson/src/ksh/arch/darwin.arm64-64/lib/lib/ast
+ dylink -e mamake -last -liconv mamake.o
[dylink] + cc -o /Users/mwilson/src/ksh/arch/darwin.arm64-64/dyn/bin/mamake mamake.o -L/Users/mwilson/src/ksh/arch/darwin.arm64-64/dyn/lib -last -liconv

# ... making src/lib/libsum ...
+ mamake -C lib/libsum install

The linker can't find libast when mamake runs.

Reverting 2e849e6 makes the build work again.

[McDutchie]

Ugh. Looks like System Integrity Protection was ramped up further on that OS, so the dynamicallly run mamake can no longer run. I am not running that OS yet to test any additional trickery needed to preserve DYLD_LIBRARY_PATH, so for now I may just have to revert 2e849e677f54fa0adb84046c7f34af322122ee08.

[McDutchie]

By the way: after reverting 2e849e677f54fa0adb84046c7f34af322122ee08, deleting arch/darwin.arm64-64/dyn/bin/mamake, and completing the build, does bin/package use work for you on macOS 13.6.6? Is it correctly invoking the dynamically linked ksh?

[posguy99]

It appears to work. bin/package test passes and it launches.

# src/Mamfile: 16-20: make announce_build_done
# KornShell Version AJM 93u+m/1.1.0-alpha+a707ef76 2024-03-30
# Type 'bin/package use' to try it out
package: test done at Sat Mar 30 20:55:49 PDT 2024 in /Users/mwilson/src/ksh-test/arch/darwin.arm64-64
[ 8:55 PM][ttys000 +1][~/src/ksh-test][dev@a707ef76]
[1006] mbp-m2 $ bin/package use
package: You are now entering a new environment set up to use the package in:
package:     /Users/mwilson/src/ksh-test/arch/darwin.arm64-64
package: You're using the shell:
package:     /Users/mwilson/src/ksh-test/arch/darwin.arm64-64/dyn/bin/ksh
package: Type 'exit' to leave and go back to normal.
Welcome to ksh 93u+m/1.1.0-alpha+a707ef76 2024-03-30 on mbp-m2 (ttys000)
- Have a look at ~ast/etc/kshrc to see all the cool stuff activated here
- Type 'builtin' for a list of built-in commands (with and without path)
- Autoloadable functions have been activated. Check them out in ~ast/fun
- Get help: use 'man commandname' for everything, even built-in commands
«0»1:~/src/ksh-test[dev] $ 

[posguy99]

I am not running that OS yet to test any additional trickery needed to preserve DYLD_LIBRARY_PATH, so for now I may just have to revert 2e849e6.

I've had decent luck running later macOS on earlier via UTM.

[McDutchie]

I've had decent luck running later macOS on earlier via UTM.

Thanks for the tip. Unfortunately, I'm not having any luck with that on UTM 4.4.5:

[posguy99]

When we were testing https://github.com/ksh93/ksh/issues/596 I installed Monterey under UTM using the IPSW and then upgraded it to Ventura.

[McDutchie]

When we were testing #596 I installed Monterey under UTM using the IPSW and then upgraded it to Ventura.

Thanks. Unfortunately, same results with that IPSW, and with both UTM 4.4.5 and UTM 4.5.1 beta. I simply don't have time to mess with this any further.

[posguy99]

Seems like reverting the change is the easiest. Is there anything gained by mamake being dynamically linked anyway?

[McDutchie]

Other than it being another test case for dynamic linking, not really.

[McDutchie]

But maybe working around it is not so hard either. SIP removes DYLD_LIBRARY_PATH from the environment, but since recursive mamake invocations are done via the shell, it's pretty easy to reinstate it as we also export Linux/BSD's LD_LIBRARY_PATH regardless of the system in use. Does this patch work on your end?

diff --git a/bin/package b/bin/package
index 4a3b7aae5..1821c1126 100755
--- a/bin/package
+++ b/bin/package
@@ -2504,6 +2504,7 @@ checkaout()   # cmd ...
                do  test -d $PACKAGEROOT/$j || $exec mkdir $PACKAGEROOT/$j || return
                done
            fi
+           rm -f "$INSTALLROOT/bin/$i" "$INSTALLROOT/dyn/bin/$i" "$INSTALLROOT/src/lib/libast/$i"
            $exec $CC $CCFLAGS -o $INSTALLROOT/bin/$i $INITROOT/$i.c || return
            test -f $i.o && $exec rm -f $i.o
            hash -r
diff --git a/src/cmd/INIT/mamake.c b/src/cmd/INIT/mamake.c
index 69c6c47e5..93ed2589c 100644
--- a/src/cmd/INIT/mamake.c
+++ b/src/cmd/INIT/mamake.c
@@ -2228,6 +2228,10 @@ static int update(Rule_t *r)
    }

    /* do */
+#if _DARWIN_C_SOURCE
+   /* System Integrity Protection workaround for pre-installed dynamically linked mamake */
+   append(buf, "DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH ");
+#endif
    append(buf, getval(state.vars, "MAMAKE"));
    append(buf, " -C ");
    append(buf, r->name);

[posguy99]

No, same fault in same place.

# src/lib/libast/Mamfile: 5075-5080: make ${INSTALLROOT}/dyn/bin/mamake
+ LDFLAGS='  '
+ export LDFLAGS
+ cat /Users/mwilson/src/ksh-test/arch/darwin.arm64-64/lib/lib/ast
+ dylink -e mamake -last -liconv mamake.o
[dylink] + cc -o /Users/mwilson/src/ksh-test/arch/darwin.arm64-64/dyn/bin/mamake mamake.o -L/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/dyn/lib -last -liconv

# ... making src/lib/libsum ...
+ mamake -C lib/libsum install
dyld[71911]: Library not loaded: libast.6.dylib
  Referenced from: <C0A72031-F543-3343-990F-D0408D8A21CA> /Users/mwilson/src/ksh-test/arch/darwin.arm64-64/dyn/bin/mamake
  Reason: tried: 'libast.6.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibast.6.dylib' (no such file), 'libast.6.dylib' (no such file), '/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file), '/Users/mwilson/src/ksh-test/arch/darwin.arm64-64/src/libast.6.dylib' (no such file)
mamake: *** exit code 134 making lib/libsum
mamake: *** exit code 1 making all
package: make failed at Tue Apr  2 14:48:57 PDT 2024 in /Users/mwilson/src/ksh-test/arch/darwin.arm64-64

[McDutchie]

Thanks. Revert it is then.