macOS: System image file fails consistency check on binary startup

[ghyatzo]

Upon creating an app, the process goes on smoothly, untill it is time to actually call the binary. I get the error: ERROR: System image file failed consistency check: maybe opened the wrong version?

Julia Version 1.8.2
Commit 36034abf260 (2022-09-29 15:21 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = 8
  JULIA_DEPOT_PATH = /Users/cshen/.local/julia

I installed julia through juliaup. the juliaup folder is in ~/.local/juliaup and the julia versions in ~/.local/julia/juliaup. the binaries in ~/.local/juliaup/bin are symlinked to ~/.local/bin which is in the PATH. I also have julia aliased to julia --project=@.

Could it be that the symlinking throws some pathing off somewhere?

[FerreolS]

Similar issue here on

Julia Version 1.8.4
Commit 00177ebc4fc (2022-12-23 21:32 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: 16 × Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 16 on 16 virtual cores
Environment:
  JULIA_NUM_THREADS = auto

the app execute smoothly when it is in the folder where it has been compiled but fail with ERROR: System image file failed consistency check: maybe opened the wrong version? when relocated anywhere else. Using the example of the package

> Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp 
  Downloaded artifact: MKL
ARGS = String[]
Base.PROGRAM_FILE = "Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp"
...

> cp -r Code/Julia/PackageCompiler.jl/examples/MyAppCompiled .
> MyAppCompiled/bin/MyApp                                     
ERROR: System image file failed consistency check: maybe opened the wrong version?

[PhilReinhold]

I'm running into this issue as well, just trying to build any sysimage, e.g.

PackageCompiler.create_sysimage(String[], sysimage_path="test_image.so")

then

julia --sysimage=test_image.so gives the same error as above.

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 12 × Apple M2 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_PKG_USE_CLI_GIT = true

[kevmoor]

Same issue, tried on Julia 1.9.3, 1.9.2, 1.10.0-beta2, all with PackageCompiler version 2.1.9, also tried on Julia 1.9.2 with PackageCompiler version 2.1.7 (the mix I had when I last had it running a few weeks ago), this versioninfo() below is from this latest combination.

Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_PKG_USE_CLI_GIT = true

[KwatMDPhD]

I'm seeing the same error when building https://github.com/KwatMDPhD/GSEA.jl/commit/79364ca00160a034f8afe1387df6107adfbdbc1a

https://github.com/comonicon/Comonicon.jl/issues/264

[sloede]

I'm seeing the same error when building KwatMDPhD/GSEA.jl@79364ca

comonicon/Comonicon.jl#264

This happens on macOS only?

[KwatMDPhD]

I have only tried on macOS Sonoma 14.0. If you have a non-mac machine, you can clone the above commit and try julia --project deps/build.lj app.

[jayscook]

I'm still getting the error on PackageCompiler 2.1.11 (which specifically addresses this issue) on macOS Ventura (13.6). Running Julia 1.8.5, not sure if 1.9 is required for the fix.

[sloede]

Thanks for reporting. Just to clarify: There is no fix available as of yet (also not in PC v2.1.11)

[jayscook]

Thanks for reporting. Just to clarify: There is no fix available as of yet (also not in PC v2.1.11)

Thanks for clarifying! I assumed that the closed issue indicated that it was fixed.

Please let me know if I can contribute to a fix in some way.

[sloede]

Please let me know if I can contribute to a fix in some way.

At the moment we are lacking an understanding of why the consistency check fails. If you manage to find out about this, we could start looking for a remedy 🙂

[KwatMDPhD]

I'm using 1.9.3

Would be great if this error can be fixed...

[ctarn]

I was able to build apps using Julia 1.9.0, PackageCompiler 2.1.7, and macOS 13. Recently I upgraded Julia to 1.9.3, PackageCompiler to 2.1.2, and macOS to 14, and the error occurred.

I tried to restore Julia to 1.9.0 and PackageCompiler to 2.1.7, and the error still occurs. So I guess macOS should cause it since Xcode Command Line Tools was also upgraded to 15, which is required by PackageCompiler to compile executables.

Unfortunately, when I tried to downgrade Xcode Command Line Tools, it warned that older version Xcode C. L. T. can not be installed on macOS 14... And I can not downgrade the OS... :(

I would recommend anyone (@jayscook ?) still using macOS 13 try to install an older version (v14.3?)of Xcode. It can be downloaded from: https://developer.apple.com/download/all/

[bjarthur]

i can't build sysimages on apple silicon either. please let me know how i can help fix it.

[hycakir]

I can confirm it is related to clang version. I have compiled MyLib with gcc from homebrew and it works.

[ctarn]

it really works! but remember to set the compiler to gcc-13 instead of just gcc since gcc is just an alias of clang by default.

you can specify the compiler by running an export in your shell:

export JULIA_CC="gcc-13"

[sloede]

Great catch! Could you please report which version of clang/gcc you are each using by default on your systems that works (using gcc --version), which version did not work, and the output of your Julia versioninfo()?

[ctarn]

clang v15 from Xcode CLT doesn't work

$ gcc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: x86_64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

(clang v14 should work, but older version Xcode CLT can not be installed on latest macOS)

gcc v13 from brew works

$ gcc-13 --version
gcc-13 (Homebrew GCC 13.2.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

versioninfo

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores
Environment:
  JULIA_CC = gcc-13

[jayscook]

Good catch!

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

$ gcc-13 --version
gcc-13 (Homebrew GCC 13.2.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_CC = gcc-13

[ctarn]

Since the versions of Julia and LLVM are different, maybe you should try older versions of gcc or upgrade Julia to 1.9.

[sloede]

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

Hm, this is unfortunate. It also means we're back to square one, since I don't understand now (again) what the necessary conditions are to trigger the error.

[hycakir]

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

julia> versioninfo()
Julia Version 1.8.5
...
  JULIA_CC = gcc-13

This is expected as per https://github.com/JuliaLang/julia/issues/49581. You should pick an earlier version of GCC/Clang or newer version of Julia.

[sloede]

@hycakir Thanks for letting us know, I didn't know about this!

@jayscook Could you please confirm that it works with gcc-12? If yes, we might want to add a note about this behavior to the docs.

[KwatMDPhD]

How can we make this work with the latest mac gcc? I would be nice to not have to downgrade.

[sloede]

How can we make this work with the latest mac gcc? I would be nice to not have to downgrade.

TBH, I do not know, since I neither know the rationale behind this check nor do I understand what it actually checks 😅 It seems this check was introduced in https://github.com/JuliaLang/julia/commit/4e7ec790b587f4fe26ce7620c83d3e3197b3c871, specifically here.

@vtjnash can you maybe shed some light on this and help us understand

• what exactly causes this error to appear when loading sysimages built with clang/gcc versions that are "too new" in some sense,
• if there is anything we can do about it on PC side, and if not
• what could be done on the Julia side to remedy this issue for building custom sysimages?

[vtjnash]

It simply checks if the symbol jl_RTLD_DEFAULT_handle_pointer exists, which is something added by codegen to the image, and that it linked to jl_RTLD_DEFAULT_handle. You may want to try setting LD_DEBUG to find out why this did not generate a completed dylib for you.

[sloede]

It simply checks if the symbol jl_RTLD_DEFAULT_handle_pointer exists, which is something added by codegen to the image, and that it linked to jl_RTLD_DEFAULT_handle. You may want to try setting LD_DEBUG to find out why this did not generate a completed dylib for you.

Thanks for the explanation! So you mean we should set LD_DEBUG when calling Julia with the broken sysimage? Can you tell us which value should we set the variable to? Also, this error does only seem to occur on macOS - there's no LD_DEBUG available IIRC, maybe you have an alternative?

Sorry for the many questions, we just don't have any enough experience to tackle this at the moment.

[vtjnash]

See man dyld for the available env options

[sloede]

I've been trying to reproduce the problem on my M1 Macbook, but without success so far - all my sysimages or apps I build seem to work.

@KwatMDPhD @bjarthur @hycakir @ctarn Can someone who had this problem please provide an MWE with a sysimage/app as small as possible and the exact commands you use to build it? In addition, please also post the output of gcc --version and Julia's versioninfo() such that we understand which environment is used exactly.

[sloede]

See man dyld for the available env options

Thanks! One more question: At the moment I seem to fail to reproduce the error on my system for both sysimages and apps. This is weird, because when using nm on any of the libjulia{"",-codegen,-internal}.dylib libraries, neither has the symbol jl_RTLD_DEFAULT_handle_pointer, and only libjulia-internal.dylib has jl_RTLD_DEFAULT_handle.

Where should this jl_RTLD_DEFAULT_handle_pointer be located and do you have an idea why it doesn't throw an error on my system although it does not exist?

[ctarn]

@sloede please see https://github.com/ctarn/MiniApp.jl/releases/tag/PC738

[sloede]

@sloede please see https://github.com/ctarn/MiniApp.jl/releases/tag/PC738

Thanks a lot, at least I can reproduce the issue now locally. However, for some reason I also get a failed consistency check when using gcc-13 🤔

[ctarn]

uh... you may need to try an older version of gcc? looking forward to it being solved. thanks for your effort.

[sloede]

uh... you may need to try an older version of gcc? looking forward to it being solved. thanks for your effort.

No, it also does not work for gcc-12. Argh... now I can reproduce the error but not the (potential) solution, thus I still don't know how to proceed 🤯

EDIT: Now I can't seem to find any combination that works - PackageCompiler.jl is completely broken on my system after updating to Xcode Command Line tools 15 :-/

[vtjnash]

jl_RTLD_DEFAULT_handle_pointer exists in the files generated by PackageCompiler and points back to the jl_RTLD_DEFAULT_handle in libjulia-internal.dylib

[sloede]

shell> nm ExampleSysimage.dylib
ExampleSysimage.dylib: no symbols

err... that does not look correct

OTOH, on Linux I get

shell> nm ExampleSysimage.so | grep jl_RTLD_DEFAULT_handle
                 U jl_RTLD_DEFAULT_handle
0000000000a8dcb8 D jl_RTLD_DEFAULT_handle_pointer

I still don't know what to make of it, while I couldn't reproduce the issue before with Xcode Command Line Tools v14, now I cannot get PackageCompiler to produce anything that works with Xcode CLT v15 😢

However, thanks to @ctarn's MWE including the resulting apps, I was able to run nm on the PREFIX/lib/julia/sys.dylib files they created and indeed:

• The failing version (with clang v15) just exports no symbols at all
• The working version (with gcc v13) exports at least something similar to the Linux output above:

  shell> nm sys.dylib | grep jl_RTLD_DEFAULT_handle
                 U _jl_RTLD_DEFAULT_handle
  0000000000e68338 S _jl_RTLD_DEFAULT_handle_pointer

@vtjnash Can you make anything of this?

[jayscook]

@hycakir Thanks for letting us know, I didn't know about this!

@jayscook Could you please confirm that it works with gcc-12? If yes, we might want to add a note about this behavior to the docs.

Having trouble getting downgraded, since gcc is provided via Xcode command line tools which are a pain to downgrade without 1. Xcode or 2. an Apple developer account, unfortunately neither of which I have available to me (work-managed dev env). Sorry! Hopefully someone else can confirm.

[hycakir]

@jayscook Can you use homebrew to install an earlier version of gcc? You may then create a symlink to this gcc in /usr/local/bin. Alternatively, you may use JULIA_CC environment variable to point to that gcc. Xcode CLT does not provide gcc as far as I know. It's just an alias to Xcode CLT clang.

@sloede I feel like your Julia is still using Xcode clang. Are you sure that JULIA_CC points to the correct gcc executable (homebrew)?

[sloede]

@sloede I feel like your Julia is still using Xcode clang. Are you sure that JULIA_CC points to the correct gcc executable (homebrew)?

Yes, I am fairly sure:

$> JULIA_CC=gcc-12 julia -q --project=.
julia> versioninfo()
Julia Version 1.9.1
Commit 147bdf428cd (2023-06-07 08:27 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 6 virtual cores
Environment:
  JULIA_CC = gcc-12

julia> run(`$(ENV["JULIA_CC"]) --version`)
gcc-12 (Homebrew GCC 12.3.0) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Process(`gcc-12 --version`, ProcessExited(0))

julia> using PackageCompiler

julia> create_sysimage(["Example"]; sysimage_path="ExampleSysimage-julia-1.9-gcc-12.dylib")
✔ [01m:23s] PackageCompiler: compiling incremental system image
-macosx_version_min has been renamed to -macos_version_min
ld: warning: ignoring duplicate libraries: '-lgcc'

julia>

$> julia -q -JExampleSysimage-julia-1.9-gcc-12.dylib
ERROR: System image file failed consistency check: maybe opened the wrong version?

[hycakir]

@sloede I tried the same thing on my macbook with the same setup but it worked out just fine for me. One thing caught my attention: I do not receive the warning that you do with -lgcc. Reading the discussions here and here, this might be the culprit for your setup. Can you try uninstalling and installing (or updating) gcc@12 as it seems to have been resolved by https://github.com/Homebrew/homebrew-core/pull/149350?

julia> versioninfo()
Julia Version 1.9.1
Commit 147bdf428cd (2023-06-07 08:27 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 6 virtual cores
Environment:
  JULIA_CC = gcc-12

[PhilReinhold]

I'll admit that I don't really understand what's going on here, but I saw this exhortation of success by adding the flags -Wl,-ld_classic, and decided to try it here.

I modified o_file_flags in PackageCompiler.jl:create_sysimg_from_object_file like so

function create_sysimg_from_object_file(object_files::Vector{String},
                                        sysimage_path::String;
                                        version,
                                        compat_level::String,
                                        soname::Union{Nothing, String})

    if soname === nothing && (Sys.isunix() && !Sys.isapple())
        soname = basename(sysimage_path)
    end
    mkpath(dirname(sysimage_path))
    # Prevent compiler from stripping all symbols from the shared lib.
    o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files -Wl,-ld_classic` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
    extra = get_extra_linker_flags(version, compat_level, soname)
    cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Base.shell_split(ldlibs())) $extra`
    run_compiler(cmd; cplusplus=true)
    return nothing
end

and now I am able to produce sysimages following the tutorial which have symbols exported and julia happily loads, despite being on xcode CLT version 15.

Probably not the right course of action to just add this flag universally, but maybe having a mechanism for passing additional linker flags could be part of the create_sysimage interface.

[DilumAluthge]

Maybe helpful: you can use the open-source xcodes command-line tool to install and switch between different Xcode versions. So you could use xcodes to install Xcode 14.

Note: xcodes is a command-line tool, and it's the one that I prefer. However, there also is a GUI version, which is called Xcodes.app.

[dokie]

I'll admit that I don't really understand what's going on here, but I saw this exhortation of success by adding the flags -Wl,-ld_classic, and decided to try it here.

I modified o_file_flags in PackageCompiler.jl:create_sysimg_from_object_file like so

function create_sysimg_from_object_file(object_files::Vector{String},
                                        sysimage_path::String;
                                        version,
                                        compat_level::String,
                                        soname::Union{Nothing, String})

    if soname === nothing && (Sys.isunix() && !Sys.isapple())
        soname = basename(sysimage_path)
    end
    mkpath(dirname(sysimage_path))
    # Prevent compiler from stripping all symbols from the shared lib.
    o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files -Wl,-ld_classic` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
    extra = get_extra_linker_flags(version, compat_level, soname)
    cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Basks for me on e.shell_split(ldlibs())) $extra`
    run_compiler(cmd; cplusplus=true)
    return nothing
end

and now I am able to produce sysimages following the tutorial which have symbols exported and julia happily loads, despite being on xcode CLT version 15.

Probably not the right course of action to just add this flag universally, but maybe having a mechanism for passing additional linker flags could be part of the create_sysimage interface.

I can confirm this works for me too on MacBook Air M1 with

Julia Version 1.9.4
Commit 8e5136fa297 (2023-11-14 08:46 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores

and

λ gcc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

But as @PhilReinhold stated, that is one heck of a hack!

[terasakisatoshi]

https://github.com/JuliaLang/PackageCompiler.jl/issues/738#issuecomment-1838901893 works on my Intel macOS :D

using PackageCompiler
using PackageCompiler: get_extra_linker_flags, julia_libdir, julia_private_libdir, ldlibs, bitflag, march, run_compiler
# https://github.com/JuliaLang/PackageCompiler.jl/issues/738#issuecomment-1838901893

# I know this causes type piracy, which is a bad solution, but it will save life for those who want to comple for macOS somehow
function PackageCompiler.create_sysimg_from_object_file(object_files::Vector{String},
                                        sysimage_path::String;
                                        version,
                                        compat_level::String,
                                        soname::Union{Nothing, String})

    if soname === nothing && (Sys.isunix() && !Sys.isapple())
        soname = basename(sysimage_path)
    end
    mkpath(dirname(sysimage_path))
    # Prevent compiler from stripping all symbols from the shared lib.
    o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files -Wl,-ld_classic` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
    extra = get_extra_linker_flags(version, compat_level, soname)
    cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Base.shell_split(ldlibs())) $extra`
    run_compiler(cmd; cplusplus=true)
    return nothing
end

using MyExe

create_app(pkgdir(MyExe), "build", force=true)

See also my gist

[justinesjw]

Hello @PhilReinhold @dokie @terasakisatoshi,

Could you please explain a little more on where should I implement these code #738

I am totally new to julia and am using it as a dependency of a dependency of a snakemake pipeline. Some clue will be very helpful!

Thanks!!

[mfbolus]

I'll admit that I don't really understand what's going on here, but I saw this exhortation of success by adding the flags -Wl,-ld_classic, and decided to try it here.

I modified o_file_flags in PackageCompiler.jl:create_sysimg_from_object_file like so

function create_sysimg_from_object_file(object_files::Vector{String},
                                        sysimage_path::String;
                                        version,
                                        compat_level::String,
                                        soname::Union{Nothing, String})

    if soname === nothing && (Sys.isunix() && !Sys.isapple())
        soname = basename(sysimage_path)
    end
    mkpath(dirname(sysimage_path))
    # Prevent compiler from stripping all symbols from the shared lib.
    o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files -Wl,-ld_classic` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
    extra = get_extra_linker_flags(version, compat_level, soname)
    cmd = `$(bitflag()) $(march()) -shared -L$(julia_libdir()) -L$(julia_private_libdir()) -o $sysimage_path $o_file_flags $(Base.shell_split(ldlibs())) $extra`
    run_compiler(cmd; cplusplus=true)
    return nothing
end

and now I am able to produce sysimages following the tutorial which have symbols exported and julia happily loads, despite being on xcode CLT version 15.

Probably not the right course of action to just add this flag universally, but maybe having a mechanism for passing additional linker flags could be part of the create_sysimage interface.

@PhilReinhold , I can also confirm your hack worked for my use case. Definitely would prefer a cleaner solution but thank all of you for your efforts and for sharing.

[SWSAmor]

Here are my tests on macOS Sonoma 14.2.1 M1 (for aarch64) with PackageCompiler v2.1.17:

clang 15.0.0 + Julia 1.9.4 = GOOD clang 15.0.0 + Julia 1.10.0 = FAILED gcc 13.2.0 + Julia 1.9.4 = GOOD gcc 13.2.0 + Julia 1.10.0 = FAILED

Based on this, I would be more inclined to think that this is an issue with Julia 1.10 rather than a difference between gcc and clang, but I am noob in this. If it helps, I can also run the same tests on M1 for x64...

[hbe72]

Brand new machine M3 (today) on Sonoma 14.3 and install from scratch.

None of options work: clang 15.0.0 + Julia 1.9.4 = FAILED clang 15.0.0 + Julia 1.10.0 = FAILED gcc 13.2.0 + Julia 1.9.4 = FAILED gcc 13.2.0 + Julia 1.10.0 = FAILED

Just observing the generated Sysimage.dylib one easily can see that it is empty. That's the reason at least for me it not passing the consistency check. Size is 17KB for a project I usually get 400MB *.dylib.

Do note that macos uses macos linker: https://github.com/Homebrew/homebrew-core/issues/17794

Check the notes for XCode 15 linker section: https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Linking

"A new linker has been written to significantly speed up static linking. It’s the default for all macOS, iOS, tvOS and visionOS binaries and anyone using the “Mergeable Libraries” feature. The classic linker can still be explicitly requested using -ld64, and will be removed in a future release."

Some discussion also here: https://www.scivision.dev/xcode-ld_classic/, but setting LDFLAGS in environment variable seem not get passed to PlatformCompiler.

Only thing which helped for me was manual modification of o_file_flags in PackageCompiler.jl:create_sysimg_from_object_file as suggested above

Hope these observations point to correct direction. Perhaps as an intermediate solution adding -Wl,-ld_classic to PackageCompiler linker flags. But if this option will be removed in future generations of XCode more permanent solution is needed.

[ghyatzo]

Only thing which helped for me was manual modification of o_file_flags in PackageCompiler.jl:create_sysimg_from_object_file as suggested above

With "helped" do you mean that it works properly when using the workaround? Or that it generated "non-empty" .dylib files but still failed the consistency check?

[hbe72]

Let me try to be clearer:

Just using different combinations of clang, gcc and julia "out-of-the-box" did not produce complete sysimage. Some combination did produce slightly larger (2MB vs 17kB), but not complete sysimage (~400MB). At the time I did not document exactly. But I can try to reproduce if needed. In all of the combinations, the compilation works fine, it takes time and compiles all the dependencies, but fails to link the complete *.dylib. And none of the produced libraries pass the consistency check.

Setting just the LDFLAGS environment variable as suggested in one of the links did not help. It does not have any impact to Julia compiler as it has to CMake based C or C++ compilation.

If I manually modify the PackageCompiler.jl as suggested at least Apple Clang 15.0.0 + Julia 1.10.0 combination works. It produces complete Sysimage and passes consistency check. I stopped there as I was happy to do some work. I can try other combinations as well, but I imagine they work as well if the culprit is the new XCode 15 linker or how it is used.

In an another machine two months back after an upgrade from MacOS 13 to 14, a Mac Mini M2, just setting JULIA_CC=gcc-13 solved the same error with Julia 1.9.3. Initially there as well the sysimage *.dylib was empty (17kB). Due to this OS upgrade, XCode was also upgraded to version 15. I can dig deeper details on this as well if needed. I do not have explanation why that works if macos ld is the root cause.

Still have access to both of the machines for further testing.

[hbe72]

FYI:

Have to add that in all of these cases below the precompilation, excution of the program and then the compilation take approximately the same time. Only thing which I see is different is the failed or working linking of dylib.

Apple Clang (15.0.0) and Julia 1.10:

This does not work out-of-the-box:

% ls -la MySysimagePrecompile.dylib
-rwxr-xr-x   1 justme  staff  16768 Jan 23 21:33 MySysimagePrecompile.dylib

% nm -g MySysimagePrecompile.dylib 
MySysimagePrecompile.dylib: no symbols

gcc-13 (13.2.0) and Julia 1.10:

This does not work out-of-the-box:

% ls -la MySysimagePrecompile.dylib
-rwxr-xr-x   1 justme  staff  2662912 Jan 23 21:36 MySysimagePrecompile.dylib

% nm -g MySysimagePrecompile.dylib 
00000000000143f0 T ___bid128_abs
00000000000201f0 T ___bid128_add
0000000000014410 T ___bid128_class
00000000000143d4 T ___bid128_copy
0000000000014400 T ___bid128_copySign
0000000000029de0 T ___bid128_div
000000000001bd54 T ___bid128_fma
000000000001f010 T ___bid128_from_int32
000000000001f040 T ___bid128_from_int64
000000000001f030 T ___bid128_from_uint32

... list continues ...

U _realloc
U _sys_icache_invalidate

Manual edit of PackageCompiler.jl w Apple Clang (15.0.0) and Julia 1.10

... that is addition of -Wl,-ld_classic to o_file_flags

This works:

% ls -la MySysimagePrecompile.dylib 
-rwxr-xr-x  1 justme  staff  450527424 Jan 23 22:12 MySysimagePrecompile.dylib

nm -g MySysimagePrecompile.dylib 
                 U ___divti3
                 U ___modti3
                 U ___udivti3
                 U ___umodti3
                 U _bzero
                 U _ijl_adopt_thread
                 U _ijl_apply_generic
                 U _ijl_boundp
                 U _ijl_bounds_error_int
                 U _ijl_bounds_error_ints
                 U _ijl_bounds_error_tuple_int
                 U _ijl_bounds_error_unboxed_int
                 U _ijl_box_char
                 U _ijl_box_float32
                 U _ijl_box_int16
                 U _ijl_box_int32
                 U _ijl_box_int64
                 U _ijl_box_ssavalue
                 U _ijl_box_uint16
                 U _ijl_box_uint32
                 U _ijl_box_uint64
                 U _ijl_checked_assignment
                 U _ijl_copy_ast
                 U _ijl_current_exception
                 U _ijl_enter_handler
                 U _ijl_error
                 U _ijl_excstack_state
                 U _ijl_field_index
                 U _ijl_gc_pool_alloc
                 U _ijl_gc_queue_root
                 U _ijl_get_binding_or_error
                 U _ijl_get_nth_field_checked
                 U _ijl_has_no_field_error
                 U _ijl_invoke
                 U _ijl_isa
                 U _ijl_lazy_load_and_lookup
                 U _ijl_load_and_lookup
                 U _ijl_new_structt
                 U _ijl_new_structv
                 U _ijl_object_id_
                 U _ijl_pop_handler
                 U _ijl_restore_excstack
                 U _ijl_subtype
                 U _ijl_throw
                 U _ijl_type_error
                 U _ijl_typeassert
                 U _ijl_undefined_var_error
                 U _jl_RTLD_DEFAULT_handle
00000000013bc3a8 S _jl_RTLD_DEFAULT_handle_pointer
                 U _jl_add_int
                 U _jl_boxed_int8_cache
                 U _jl_boxed_uint8_cache
                 U _jl_diverror_exception
                 U _jl_egal__unboxed
                 U _jl_emptytuple
                 U _jl_f__apply_iterate
                 U _jl_f__apply_pure
                 U _jl_f__call_in_world
                 U _jl_f__call_latest
                 U _jl_f__expr
                 U _jl_f__svec_ref
                 U _jl_f__typevar
                 U _jl_f_applicable
                 U _jl_f_apply_typ
                 U _jl_f_arraysize
                 U _jl_f_fieldtype
                 U _jl_f_finalizer
                 U _jl_f_getfield
                 U _jl_f_isa
                 U _jl_f_isdefined
                 U _jl_f_issubtype
                 U _jl_f_setfield
                 U _jl_f_sizeof
                 U _jl_f_svec
                 U _jl_f_tuple
                 U _jl_f_typeassert
                 U _jl_false
                 U _jl_field_isdefined_checked
00000000013c1580 D _jl_image_pointers
                 U _jl_libjulia_internal_handle
                 U _jl_neg_int
                 U _jl_not_int
                 U _jl_nothing
00000000013c15c0 D _jl_system_image_data
0000000001157118 S _jl_system_image_size
                 U _jl_true
                 U _jl_undefref_exception
                 U _jl_world_counter
                 U _jl_xor_int
                 U _julia__gnu_f2h_ieee
                 U _julia__gnu_h2f_ieee
                 U _julia__truncdfhf2
                 U _memcpy
                 U _memmove
                 U _memset
                 U _sigsetjmp