OpenSSL cross compile with macOS build and arm host

[jhgorse]

Greetings,

I was playing with meson because it seemed fast. I compiled OpenSSL in 7.5 seconds as the wrapdb subproject compared to 5:11 seconds for regular build. This inspired me to try to cross compile to 32 bit ARM linux host/target using macOS M1 as the build box.

At first I tried linaro/arm embedded toolchains, but neither support macOS to arm-linux, just macOS to baremetal arm-none. Okay, so what about clang?

cross-file.txt

[binaries]
c = 'clang'
cpp = 'clang++'
ar = 'ar'
strip = 'strip'
as = 'as'

[build_machine]
system = 'darwin'
cpu_family = 'aarch64'
cpu = 'aarch64'

[host_machine]
system = 'linux'
cpu = 'arm'
cpu_family = 'arm'
endian = 'little'

build.meson

project('tutorial', 'c', default_options : ['c_std=c11', 'cpp_std=c++11'])
ssl = subproject('openssl')
executable('demo', 'main.c')

I had to clear the openssl_libraries atomic entry in /subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/meson.build. Couldn't figure out how to override it at the top level meson file or the cross-file.

openssl_libraries = [
  'dl',
  # 'atomic',
]

We get to the point where it can't handle the assembly.

test_build % meson compile -C builddir/
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /opt/homebrew/bin/ninja -C /private/tmp/test_build/builddir
ninja: Entering directory `/private/tmp/test_build/builddir'
[898/970] Compiling C object subprojects/openssl-3.0.7/libcrypto.so.p/generated-config_archs_linux-armv4_asm_crypto_bn_armv4-gf2m.S.o
FAILED: subprojects/openssl-3.0.7/libcrypto.so.p/generated-config_archs_linux-armv4_asm_crypto_bn_armv4-gf2m.S.o
clang -Isubprojects/openssl-3.0.7/libcrypto.so.p -Isubprojects/openssl-3.0.7 -I../subprojects/openssl-3.0.7 -I../subprojects/openssl-3.0.7/include -I../subprojects/openssl-3.0.7/crypto -I../subprojects/openssl-3.0.7/crypto/modes -I../subprojects/openssl-3.0.7/crypto/ec/curve448 -I../subprojects/openssl-3.0.7/crypto/ec/curve448/arch_32 -I../subprojects/openssl-3.0.7/providers/common/include -I../subprojects/openssl-3.0.7/providers/implementations/include -Isubprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm -I../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm -I../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/include -I../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/crypto -I../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/providers/common/include -fcolor-diagnostics -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O0 -g -fPIC -pthread -Wno-missing-field-initializers -DOPENSSL_NO_HW '-DOPENSSLDIR="/etc/ssl"' '-DENGINESDIR="/dev/null"' '-DMODULESDIR="/dev/null"' -DOPENSSL_USE_NODELETE -DOPENSSL_BUILDING_OPENSSL -DAES_ASM -DBSAES_ASM -DECP_NISTZ256_ASM -DGHASH_ASM -DKECCAK1600_ASM -DOPENSSL_BN_ASM_GF2m -DOPENSSL_BN_ASM_MONT -DOPENSSL_CPUID_OBJ -DPOLY1305_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DOPENSSL_PIC -Wa,--noexecstack -MD -MQ subprojects/openssl-3.0.7/libcrypto.so.p/generated-config_archs_linux-armv4_asm_crypto_bn_armv4-gf2m.S.o -MF subprojects/openssl-3.0.7/libcrypto.so.p/generated-config_archs_linux-armv4_asm_crypto_bn_armv4-gf2m.S.o.d -o subprojects/openssl-3.0.7/libcrypto.so.p/generated-config_archs_linux-armv4_asm_crypto_bn_armv4-gf2m.S.o -c ../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/crypto/bn/armv4-gf2m.S
../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/crypto/bn/armv4-gf2m.S:7:1: error: unknown directive
.code 32
^
../subprojects/openssl-3.0.7/generated-config/archs/linux-armv4/asm/crypto/bn/armv4-gf2m.S:11:1: error: unknown directive
.type mul_1x1_ialu,%function
^

So it appears to be neglecting the clang cross compile requirements: '--target=arm-linux-gnu', '-mfloat-abi=hard'

When I re-run the above failed compile command with the target and mfloat defined, it succeeds. I tried appending these as c_args to the executable without luck.

How do I get this to compile?

Cheers, Joe

[jhgorse]

Switched to gcc. I was able to build OpenSSL 3.0.7 in 10 seconds using the armv7-unknown-linux-gnueabihf-gcc-11.2.0 from here: https://github.com/messense/homebrew-macos-cross-toolchains/blob/main/armv7-unknown-linux-gnueabihf.rb

armv7-unknown-linux-gnueabihf-gcc-11.2.0 -v
Using built-in specs.
COLLECT_GCC=armv7-unknown-linux-gnueabihf-gcc-11.2.0
COLLECT_LTO_WRAPPER=/private/tmp/armv7-unknown-linux-gnueabihf/bin/../libexec/gcc/armv7-unknown-linux-gnueabihf/11.2.0/lto-wrapper
Target: armv7-unknown-linux-gnueabihf
Configured with: /Volumes/build/.build/armv7-unknown-linux-gnueabihf/src/gcc/configure --build=aarch64-build_apple-darwin21.3.0 --host=aarch64-build_apple-darwin21.3.0 --target=armv7-unknown-linux-gnueabihf --prefix=/Volumes/tools/armv7-unknown-linux-gnueabihf --exec_prefix=/Volumes/tools/armv7-unknown-linux-gnueabihf --with-sysroot=/Volumes/tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot --enable-languages=c,c++,fortran --with-fpu=neon-vfpv4 --with-float=hard --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libsanitizer --disable-libmpx --with-gmp=/Volumes/build/.build/armv7-unknown-linux-gnueabihf/buildtools --with-mpfr=/Volumes/build/.build/armv7-unknown-linux-gnueabihf/buildtools --with-mpc=/Volumes/build/.build/armv7-unknown-linux-gnueabihf/buildtools --with-isl=/Volumes/build/.build/armv7-unknown-linux-gnueabihf/buildtools --enable-lto --enable-threads=posix --enable-target-optspace --enable-plugin --enable-gold --disable-nls --disable-multilib --with-local-prefix=/Volumes/tools/armv7-unknown-linux-gnueabihf/armv7-unknown-linux-gnueabihf/sysroot --enable-long-long
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 11.2.0 (GCC)

Added this to the stock wrapdb OpenSSL meason.build defines += ['_BSD_SOURCE']

meson.build

project('tutorial', 'c')
ssl = subproject('openssl')
thread_dep = dependency('threads')
executable('demo', 'main.c', dependencies : thread_dep)

cross-file.txt

[constants]
arch = 'armv7-unknown-linux-gnueabihf'

[binaries]
c = arch + '-gcc'
cpp = arch + '-g++'
ar = arch + '-ar'
strip = arch + '-strip'
as = arch + '-as'
c_ld = 'gold'

[build_machine]
system = 'darwin'
cpu_family = 'aarch64'
cpu = 'aarch64'

[host_machine]
system = 'linux'
cpu = 'arm'
cpu_family = 'arm'
endian = 'little'

Pretty nice.

I am going to switch back to clang for a moment to see if I can't get over the macos link to elf issue.

[jhgorse]

openssl| Message: Generating OpenSSL configs... Only works on Linux, they say. They must have been cached in the armv7-unknown-linux-gnueabihf-gcc-11.2.0 build, though apparently the OpenSSL configs were there in 32 bit arm assembly.

cpanm Text::Template and following the flow of generator.sh and we arrive at generated headers on macos. Notably, there is a no-asm variant for the embedded arm we are targeting, in case we want to pivot.

Shockingly, this fails sometimes:

find config/archs -iname '*.s' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'

cp config/archs/linux-x86_64/asm/crypto/aes/aesni-sha1-x86_64.s ../../../generated-config/archs/linux-x86_64/asm/crypto/aes/aesni-sha1-x86_64.s
cp config/archs/linux-x86_64/asm/crypto/aes/aesni-sha256-x86_64.s ../../../generated-%

Substituting gxargs for xargs from brew findutils does the trick.

meson does not respect the assembler choice of nasm in this case. Failing to assembly .S file with clang. Oh yeah, subprojects override. nasm_args is also not useful, as it is referred to in cross-file.txt but not used. Simply adding '--target=arm-linux-gnu', '-mfloat-abi=hard' to c_args confuses meson and then it starts trying to connect to the darwin subsystem. Yikes.

Hardcode asm = 'no-asm' for OpenSSL subproject.

I am confusing the linker logic. Related to https://github.com/mesonbuild/meson/issues/6662 c_ld = 'lld' (system clang lld) results in

/opt/homebrew/opt/llvm/bin/clang  -o subprojects/openssl-3.0.7/libcrypto.so ... snip ...
-Wl,--as-needed -Wl,--no-undefined -fuse-ld=lld -shared -fPIC -Wl,--start-group -Wl,-soname,libcrypto.so -Wl,--end-group -pthread
ld64.lld: error: unknown argument '--as-needed'
ld64.lld: error: unknown argument '--no-undefined'
ld64.lld: error: unknown argument '--start-group'
ld64.lld: error: unknown argument '-soname'
ld64.lld: error: unknown argument '--end-group'
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
[920/957] Compiling C object subprojects/openssl-3.0.7/libssl.so.p/ssl_ssl_ciph.c.o
ninja: build stopped: subcommand failed.

Removing the associated unknown arguments manually leads to

ld64.lld: error: duplicate symbol: _bind_engine
>>> defined in e_capi.c:1916 (../subprojects/openssl-3.0.7/engines/e_capi.c:1916)
>>>            subprojects/openssl-3.0.7/libcrypto.so.p/engines_e_capi.c.o
>>> defined in e_padlock.c:746 (../subprojects/openssl-3.0.7/engines/e_padlock.c:746)
>>>            subprojects/openssl-3.0.7/libcrypto.so.p/engines_e_padlock.c.o
and 3 others

Changing to homebrew lld.

lld is a generic driver.
Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
[919/957] Compiling C object subprojects/openssl-3.0.7/libssl.so.p/ssl_s3_lib.c.o

c_ld = homebrew_clang_path + ld.lld

ld.lld: error: unknown argument '-dynamic', did you mean '-Bdynamic'
ld.lld: error: unknown argument '-dylib'
ld.lld: error: unknown argument '-arch'
ld.lld: error: unknown argument '-platform_version'
ld.lld: error: unknown argument '-syslibroot'
ld.lld: error: unable to find library -lto_library
ld.lld: error: /opt/homebrew/Cellar/llvm/15.0.6/lib/libLTO.dylib: unknown file type
ld.lld: error: cannot open arm64: No such file or directory
ld.lld: error: cannot open macos: No such file or directory
ld.lld: error: cannot open 12.0.0: No such file or directory
ld.lld: error: cannot open 12.0.0: No such file or directory
ld.lld: error: cannot open /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk: Is a directory
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_cbc.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_cfb.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_core.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_ecb.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_ige.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_misc.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_ofb.c.o: unknown file type
ld.lld: error: subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_wrap.c.o: unknown file type
ld.lld: error: too many errors emitted, stopping now (use --error-limit=0 to see all errors)
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
[921/957] Compiling C object subprojects/openssl-3.0.7/libssl.so.p/ssl_ssl_ciph.c.o
ninja: build stopped: subcommand failed.

The armv7-unknown-linux-gnueabihf-ldd, ld, ld.gold are not terribly interesting, though they should be able to work. Some complain about not being able to read the object files, which means meson was being naughty and building mach-o objects when they should have been elf.

file subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_cbc.c.o
subprojects/openssl-3.0.7/libcrypto.so.p/crypto_aes_aes_cbc.c.o: Mach-O 64-bit object arm64

That's enough for today. Meson is interesting to me because it is fast. I do not see that the internals can be exposed with any sort of trace log. I'll need to set up a dev env to see what's going on inside.

[eli-schwartz]

@nazar-pc did basically all the work on the openssl subproject wrap, and perhaps knows a bit more about things such as running the generator there.

[eli-schwartz]

Substituting gxargs for xargs from brew findutils does the trick.

A more portable approach would be to ask "why are we using xargs here anyway". :p

diff --git a/subprojects/packagefiles/openssl/generator.sh b/subprojects/packagefiles/openssl/generator.sh
index f8993006..0c9e3a4c 100755
--- a/subprojects/packagefiles/openssl/generator.sh
+++ b/subprojects/packagefiles/openssl/generator.sh
@@ -31,11 +31,12 @@ rm -rf config/archs
 LANG=C make -C config

 # Copy generated files back into correct place
-find config/archs -name 'meson.build' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.asm' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.c' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.h' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -iname '*.s' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
+cmd='mkdir -p ../../../generated-$(dirname "$1"); cp "$1" ../../../generated-"$1"'
+find config/archs -name 'meson.build' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.asm' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.c' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.h' -exec sh -c "$cmd" {} \;
+find config/archs -iname '*.s' -exec sh -c "$cmd" {} \;

 # AIX is not supported by Meson
 rm -rf ../../../generated-config/archs/aix*

[eli-schwartz]

I do not see that the internals can be exposed with any sort of trace log. I'll need to set up a dev env to see what's going on inside.

It's all python, so you can just git clone the mesonbuild repo, add print() statements wherever you please, and run it as path/to/git/checkout/meson.py setup srcdir/ builddir/.

There's a fair amount of debug info logged to builddir/meson-logs/meson-log.txt, including the commands used perform compiler/linker detection, the contents/results of compiler probes / has_function / find_library etc. -- this may include the information you are interested in.

[jhgorse]

@nazar-pc did basically all the work on the openssl subproject wrap, and perhaps knows a bit more about things such as running the generator there.

@nazar-pc did well with node's out-of-tree OpenSSL build. If you are ever in my neck of the woods (DC), reach out, I'd like to buy you a beer. Cheers! =)

I agree. xargs is to be avoided except for in the moment hacking. Though xargs is slightly more intuitive than find's weird but portable -exec {} \; syntax.

[jhgorse]

meson-log.txt is good.

meson-log.txt Click me

``` Build started at 2022-12-18T09:37:30.519383 Main binary: /opt/homebrew/opt/python@3.10/bin/python3.10 Build Options: --cross-file=x86_64-armv7-linux.txt --native-file=native.txt Python system: Darwin The Meson build system Version: 1.0.0.rc2 Source dir: /Users/jhg/tmp/test_build Build dir: /Users/jhg/tmp/test_build/builddir Build type: cross build Project name: tutorial Project version: undefined ----- Detecting compiler via: /opt/homebrew/opt/llvm/bin/clang --version compiler returned compiler stdout: Homebrew clang version 15.0.6 Target: arm64-apple-darwin21.6.0 Thread model: posix InstalledDir: /opt/homebrew/opt/llvm/bin compiler stderr: Running command: /opt/homebrew/opt/llvm/bin/clang -E -dM - ----- Detecting linker via jhg: /opt/homebrew/opt/llvm/bin/clang -Wl,--version -fuse-ld=/opt/homebrew/opt/llvm/bin/ld.lld linker returned linker stdout: Homebrew LLD 15.0.6 (compatible with GNU linkers) linker stderr: ld.lld: error: unknown argument '-dynamic', did you mean '-Bdynamic' ld.lld: error: unknown argument '-arch' ld.lld: error: unknown argument '-platform_version' ld.lld: error: unknown argument '-syslibroot' clang-15: error: linker command failed with exit code 1 (use -v to see invocation) LLVMDynamicLinker: compiler=['/opt/homebrew/opt/llvm/bin/clang'] for_machine= comp_class.LINKER_PREFIX='-Wl,' override=['-fuse-ld=/opt/homebrew/opt/llvm/bin/ld.lld'] v='15.0.6' Sanity testing C compiler: /opt/homebrew/opt/llvm/bin/clang Is cross compiler: True. Sanity check compiler command line: /opt/homebrew/opt/llvm/bin/clang sanitycheckc.c -o sanitycheckc_cross.exe -D_FILE_OFFSET_BITS=64 -c Sanity check compile stdout: ----- Sanity check compile stderr: ----- C compiler for the host machine: /opt/homebrew/opt/llvm/bin/clang (clang 15.0.6 "Homebrew clang version 15.0.6") C linker for the host machine: /opt/homebrew/opt/llvm/bin/clang ld.lld 15.0.6 detecting CPU family based on trial='arm64' ----- Detecting compiler via: /opt/homebrew/opt/llvm/bin/clang --version compiler returned compiler stdout: Homebrew clang version 15.0.6 Target: arm64-apple-darwin21.6.0 Thread model: posix InstalledDir: /opt/homebrew/opt/llvm/bin compiler stderr: Running command: /opt/homebrew/opt/llvm/bin/clang -E -dM - ----- Detecting linker via jhg: /opt/homebrew/opt/llvm/bin/clang -Wl,--version -fuse-ld=/opt/homebrew/opt/llvm/bin/lld linker returned linker stdout: linker stderr: lld is a generic driver. Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead clang-15: error: linker command failed with exit code 1 (use -v to see invocation) ----- Detecting Apple linker via: /opt/homebrew/opt/llvm/bin/clang -Wl,-v linker stdout: linker stderr: @(#)PROGRAM:ld PROJECT:ld64-819.6 BUILD 14:58:44 Aug 5 2022 configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em Library search paths: /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib Framework search paths: /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/ Undefined symbols for architecture arm64: "_main", referenced from: implicit entry/start for main executable ld: symbol(s) not found for architecture arm64 clang-15: error: linker command failed with exit code 1 (use -v to see invocation) Sanity testing C compiler: /opt/homebrew/opt/llvm/bin/clang Is cross compiler: False. Sanity check compiler command line: /opt/homebrew/opt/llvm/bin/clang sanitycheckc.c -o sanitycheckc.exe -fuse-ld=/opt/homebrew/opt/llvm/bin/lld Sanity check compile stdout: ----- Sanity check compile stderr: lld is a generic driver. Invoke ld.lld (Unix), ld64.lld (macOS), lld-link (Windows), wasm-ld (WebAssembly) instead clang-15: error: linker command failed with exit code 1 (use -v to see invocation) ----- Compiler for language c for the build machine not found. detecting CPU family based on trial='arm64' detecting CPU family based on trial='arm64' Build machine cpu family: aarch64 Build machine cpu: aarch64 Host machine cpu family: arm Host machine cpu: arm Target machine cpu family: arm Target machine cpu: arm Executing subproject openssl Project name: openssl Project version: 3.0.7 C compiler for the host machine: /opt/homebrew/opt/llvm/bin/clang (clang 15.0.6 "Homebrew clang version 15.0.6") C linker for the host machine: /opt/homebrew/opt/llvm/bin/clang ld.lld 15.0.6 detecting CPU family based on trial='arm64' C compiler for the build machine: /opt/homebrew/opt/llvm/bin/clang (clang 15.0.6 "Homebrew clang version 15.0.6") C linker for the build machine: /opt/homebrew/opt/llvm/bin/clang ld64 819.6 detecting CPU family based on trial='arm64' detecting CPU family based on trial='arm64' Run-time dependency threads found: YES Program as found: YES Message: OpenSSL is configured without ASM support detecting CPU family based on trial='arm64' detecting CPU family based on trial='arm64' detecting CPU family based on trial='arm64' Build targets in project: 3 Subproject openssl finished. Dependency threads found: YES unknown (cached) detecting CPU family based on trial='arm64' Build targets in project: 4 tutorial undefined Subprojects openssl : YES User defined options Cross files : x86_64-armv7-linux.txt Native files: native.txt Found ninja-1.11.1 at /opt/homebrew/bin/ninja Running compile: Working directory: /Users/jhg/tmp/test_build/builddir/meson-private/tmpk7t7kcgc Command line: /opt/homebrew/opt/llvm/bin/clang /Users/jhg/tmp/test_build/builddir/meson-private/tmpk7t7kcgc/testfile.c -o /Users/jhg/tmp/test_build/builddir/meson-private/tmpk7t7kcgc/output.obj -c -D_FILE_OFFSET_BITS=64 -O0 -Werror=implicit-function-declaration -Werror=unknown-warning-option -Werror=unused-command-line-argument -Werror=ignored-optimization-argument --print-search-dirs Code: Compiler stdout: programs: =/opt/homebrew/opt/llvm/bin:/opt/homebrew/Cellar/llvm/15.0.6/bin libraries: =/opt/homebrew/Cellar/llvm/15.0.6/lib/clang/15.0.6 Compiler stderr: ```

I see. homebrew clang is bringing its build machine flags to the link.

% /opt/homebrew/opt/llvm/bin/clang -Wl,--version -fuse-ld=/opt/homebrew/opt/llvm/bin/ld.lld -v
Homebrew clang version 15.0.6
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
 "/opt/homebrew/opt/llvm/bin/ld.lld" -demangle -lto_library /opt/homebrew/Cellar/llvm/15.0.6/lib/libLTO.dylib -dynamic -arch arm64 -platform_version macos 12.0.0 12.0.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -o a.out --version -lSystem /opt/homebrew/Cellar/llvm/15.0.6/lib/clang/15.0.6/lib/darwin/libclang_rt.osx.a
ld.lld: error: unknown argument '-dynamic', did you mean '-Bdynamic'
ld.lld: error: unknown argument '-arch'
ld.lld: error: unknown argument '-platform_version'
ld.lld: error: unknown argument '-syslibroot'
Homebrew LLD 15.0.6 (compatible with GNU linkers)
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)

If we spec the cross compile target/host --target=arm-linux-gnu -mfloat-abi=hard then things look better.

% /opt/homebrew/opt/llvm/bin/clang -Wl,--version -fuse-ld=/opt/homebrew/opt/llvm/bin/ld.lld --target=arm-linux-gnu -mfloat-abi=hard -v
Homebrew clang version 15.0.6
Target: arm-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
 "/opt/homebrew/opt/llvm/bin/ld.lld" --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -pie -EL -X --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o a.out Scrt1.o crti.o crtbeginS.o -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib/../lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib --version -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed crtendS.o crtn.o
Homebrew LLD 15.0.6 (compatible with GNU linkers)

I see this occurs at detect.py which runs twice. Perhaps once for build and once for host machines.

It seems to understand that I want arm 32 bit linux for target/host, but it only currently cares about the build machine's arm64 for the subproject based on detecting CPU family based on trial='arm64' which is pulled from python's platform.machine(). Not cross-file.txt. Interesting.

Removed OpenSSL subproject for now. Returning to hello world clang cross compile. It fails. Clang is not specifying --target=arm-linux-gnu -mfloat-abi=hard.

I saw some possibly related issues in the history of meson, but nothing current.

Why would the clang cross handler not listen to the cross-file?

[jhgorse]

Substituting gxargs for xargs from brew findutils does the trick.

A more portable approach would be to ask "why are we using xargs here anyway". :p

diff --git a/subprojects/packagefiles/openssl/generator.sh b/subprojects/packagefiles/openssl/generator.sh
index f8993006..0c9e3a4c 100755
--- a/subprojects/packagefiles/openssl/generator.sh
+++ b/subprojects/packagefiles/openssl/generator.sh
@@ -31,11 +31,12 @@ rm -rf config/archs
 LANG=C make -C config

 # Copy generated files back into correct place
-find config/archs -name 'meson.build' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.asm' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.c' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -name '*.h' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
-find config/archs -iname '*.s' | xargs -I % sh -c 'mkdir -p ../../../generated-$(dirname %); cp % ../../../generated-%'
+cmd='mkdir -p ../../../generated-$(dirname "$1"); cp "$1" ../../../generated-"$1"'
+find config/archs -name 'meson.build' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.asm' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.c' -exec sh -c "$cmd" {} \;
+find config/archs -name '*.h' -exec sh -c "$cmd" {} \;
+find config/archs -iname '*.s' -exec sh -c "$cmd" {} \;

 # AIX is not supported by Meson
 rm -rf ../../../generated-config/archs/aix*

I am not sure why, but the above approach doesn't quite work in bash.

bash-5.1$ cmd='echo mkdir -p ../../../generated-$(dirname "$1"); echo cp "$1" ../../../generated-"$1"'
bash-5.1$ find config/archs -name 'meson.build' -exec sh -c "$cmd" {} \;
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-
mkdir -p ../../../generated-.
cp  ../../../generated-

This works in bash:

cmd () {
  mkdir -p ../../../generated-$(dirname "$1")
  cp "$1" ../../../generated-"$1"
}
export -f cmd
find config/archs -name 'meson.build' -exec sh -c "cmd {}" \;
find config/archs -name '*.asm' -exec sh -c "cmd {}" \;
find config/archs -name '*.c' -exec sh -c "cmd {}" \;
find config/archs -name '*.h' -exec sh -c "cmd {}" \;
find config/archs -iname '*.s' -exec sh -c "cmd {}" \;

Apparently embedding functions like this may be frowned upon due to Shellshock. It is intentionally not supported in zsh in favor of embedding functions into dotfiles, which misses the temporary script use case that we use. https://unix.stackexchange.com/questions/59360/what-is-the-zsh-equivalent-of-bashs-export-f

I'll submit the PR for that.

[eli-schwartz]

I am not sure why, but the above approach doesn't quite work in bash.

Because I forgot to specify it as -exec sh -c "$cmd" _ignored {} \;

The first argument to the shell is of course $0 rather than $1.

And using -exec sh -c "cmd {}" instead is unsafe -- because {} will not be passed as a single element -- but that's more of a theoretical issue than a practical one, since the result filenames probably don't ever include whitespace or quote characters embedded in them. :D

[jhgorse]

Interesting. Put in the ignored argument to fill $0. I thought $0 was supposed to be the script name. Weird.

The proposed change does work. I think that the concern about paths with spaces can be addressed with single quotes'{}'.

jhg@jhg-mbp21 openssl % bash
cmd='echo mkdir -p ../../../generated-$(dirname "$1"); echo cp "$1" ../../../generated-"$1"'
bash-5.1$ find config/archs -name 'meson.build' -exec sh -c "$cmd" _ignored '{}' \;
mkdir -p ../../../generated-config/archs/linux-elf/asm_avx2
cp config/archs/linux-elf/asm_avx2/meson.build ../../../generated-config/archs/linux-elf/asm_avx2/meson.build

I'll amend the PR.

[neheb]

generator.sh should probably be converted to use python.

[eli-schwartz]

I thought $0 was supposed to be the script name. Weird.

Sure, but in this case the script name is _ignored. We don't care that ps or htop will report that as the script name for a sh -c ....

I think that the concern about paths with spaces can be addressed with single quotes '{}'.

Nope. You're single-quoting and safely passing the {} argument to find, but it was always safe to begin with.

The only technically pure way to dynamically build a shell script from a string is:

• pass dynamic arguments as parameters to the script and reference them by quoting "$1", or
• using some complicated argument escaping function that runs through a list of shell metacharacters, and replaces each one with suitable escape sequences. The list of shell metacharacters is different and mutually incompatible depending on whether you use single quotes, double quotes, or no quotes. Using single quotes is NEVER sufficient on its own

Programming languages such as python have trivial routines to do the latter. But the former is so simple I usually just do that. Which is why I recommended it.

As I said, it's not super important if you know that the openssl source tree doesn't contain filenames with metacharacters. I'm just a shell script nerd and I enjoy pontificating about the subject.

[jhgorse]

generator.sh should probably be converted to use python.

I see. A python script would maintain consistency with the original meson python3 requirement. Introduce fewer dependencies and tools and minimize exposure to inconsistencies.

On the other hand, I am not confident that my python script substitute would be 52 lines or less and as readable as this bash script.

This might be a personal problem regarding my ability with python. =)

[jhgorse]

@eli-schwartz Interesting. I had dug deep into bash years ago, maybe 2000's or so, due to a string escaping problem which turned out to be unsolvable in bash. I did not get to the absolute bottom of it but came away with the practical understanding that not all string manipulation can or should be done by the shell.

Thank you for taking the time to pontificate. I think this stuff is fun. Especially when it works. =)

[eli-schwartz]

Yup, that's exactly why it's best to create it as an argv variable -- it's guaranteed to be defined correctly without escaping problems, and once you have a variable already, it's easy to just make sure that the variable is preserved. Simple double quotes of an existing variable will preserve its value without any fuss, because you're not manipulating anything.

[eli-schwartz]

(If you cannot do that, then I prefer embedding single-quoted strings. The trick there is that you need to un-quote a literal single quote, and then escape it as \', which means that give a string like it's a string you can shell-escape it as 'it'\''s a string'. To put it another way, take the string, replace any existing single quote with the sequence '\'', and then wrap it in an additional layer of quotes. This makes use of the fact that inside single quoted strings, the only metacharacter is the single-quote character.)

[neheb]

generator.sh should probably be converted to use python.

I see. A python script would maintain consistency with the original meson python3 requirement. Introduce fewer dependencies and tools and minimize exposure to inconsistencies.

Maybe also adds Windows support :).

On the other hand, I am not confident that my python script substitute would be 52 lines or less and as readable as this bash script.

On that note, pylint is a great help. Turns out the script I wrote for openal-soft is subtly broken :(.

This might be a personal problem regarding my ability with python. =)

[nazar-pc]

Maybe also adds Windows support :).

Not really, the complexity with Windows and macOS isn't just shell.