Library architecture confusion when running on ARM M1

[tamasgal]

I am running Julia 1.6.2 (x64 mode with Rosetta 2) and having problems with PackageCompiler.jl.

I get warnings like attempting to link with file built for macOS-x86_64 and when using the generated sysimage.so, it tries to load sysimage.so.dylib which is not present. I don't know why it puts .dylib at the end (which would be the standard library file extension on macOS).

Btw. it was working fine a few days ago and I have no idea what happened or when it eventually broke...

Here is how I build and use the sysimage (which still works nicely on the CI using Linux on x86):

julia --project -e 'using PackageCompiler; create_sysimage(["CSV", "Colors", "DataFrames", "Distributions", "DocOpt", "DrWatson", "HDF5", "JSON", "LaTeXStrings", "LandauDistribution", "LsqFit", "NeRCA", "PGFPlotsX", "Query", "StatsBase"]; sysimage_path="sysimage.so", precompile_statements_file="scripts/precompile.jl")'
[ Info: PackageCompiler: creating system image object file, this might take a while...
ld: warning: ignoring file /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /Applications/Julia-1.6.app/Contents/Resources/julia/lib/libjulia.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /var/folders/84/mcvklq757tq1nfrkbxvvbq8m0000gn/T/jl_iRqtyC.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
julia -J sysimage.so --project plots/cherenkov_path_length_vs_nphe.jl -o plots
ERROR: could not load library "/Users/tamasgal/Dev/PhD/sysimage.so"
dlopen(/Users/tamasgal/Dev/PhD/sysimage.so.dylib, 6): image not found
make[1]: *** [plots/cherenkov_path_length_vs_nphe.pdf] Error 1
make: *** [all] Error 2

I then changed the name of the system image from sysimage.so to sysimage.dylib which got rid of the file-naming issue, but then I get a wrong architecture message since it attempts to build for macOS-arm64 and link with macOS-x86_64:

░ tamasgal@silentbox:PhD  master ●● 🅒 base took 7s
░ 16:36:20 > make
/Applications/Xcode.app/Contents/Developer/usr/bin/make phd_thesis.pdf
make install
julia --project -e "using Pkg; Pkg.resolve()"
  No Changes to `~/Dev/PhD/Project.toml`
  No Changes to `~/Dev/PhD/Manifest.toml`
julia --project -e "using Pkg; Pkg.instantiate()"
julia --project -e "using Pkg; Pkg.precompile()"
julia --project -e 'using PackageCompiler; create_sysimage(["CSV", "Colors", "DataFrames", "Distributions", "DocOpt", "DrWatson", "HDF5", "JSON", "LaTeXStrings", "LandauDistribution", "LsqFit", "NeRCA", "PGFPlotsX", "Query", "StatsBase"]; sysimage_path="sysimage.dylib", precompile_statements_file="scripts/precompile.jl")'
[ Info: PackageCompiler: creating system image object file, this might take a while...
ld: warning: ignoring file /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /Applications/Julia-1.6.app/Contents/Resources/julia/lib/libjulia.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /var/folders/84/mcvklq757tq1nfrkbxvvbq8m0000gn/T/jl_1FhnNx.o, building for macOS-arm64 but attempting to link with file built for unknown-x86_64
julia -J sysimage.dylib --project plots/cherenkov_path_length_vs_nphe.jl -o plots
ERROR: could not load library "/Users/tamasgal/Dev/PhD/sysimage.dylib"
dlopen(/Users/tamasgal/Dev/PhD/sysimage.dylib, 6): no suitable image found.  Did find:
    /Users/tamasgal/Dev/PhD/sysimage.dylib: mach-o, but wrong architecture
    /Users/tamasgal/Dev/PhD/sysimage.dylib: mach-o, but wrong architecture
make[1]: *** [plots/cherenkov_path_length_vs_nphe.pdf] Error 1
make: *** [all] Error 2

I am a bit puzzled why this worked before and not now, so asking for help. Maybe someone has a hint how to deal with this arch-issue ;)

I still cannot use Julia 1.7 for ARM (M-series) since a few crucial packages are not available due to missing binary dependencies (e.g. HDF5.jl)

[KristofferC]

I currently don't have an M1 machine so I cannot really debug and test this.

[schlichtanders]

@tamasgal I am probably also running in an ARM compatibility problem. Have you found some solutions for your case=

I guess the default compile options for the sysimage are leading to incorrect behaviour when run on ARM. (I am running on Ampere Altra). My error is however completely different:

exec /usr/local/julia/bin/julia: exec format error

nevertheless, maybe you have some tips what to try for getting sysimages on ARM to work

[tamasgal]

Sorry I don't have any hint. I gave up after a while and did not use sysimages on my M1.

Actually with Julia 1.8 and 1.9-rc* I am very happy with the load times (for development, prototyping and high-level analysis) and for HPC I don't need M1 sysimages anyways.

[schlichtanders]

@KristofferC do you have some general tips about which compile flags to use for compiling sysimages which are arm friendly?