Precompiling on Mac (M-series): 'could not load library' and 'Platform `arm64-apple-darwin22.4.0` is not an officially supported platform' errors

[pbouffard]

This is a bit of a Heisenbug--if a bug at all and even if so possibly not a JuliaLang but I want to capture this somewhere other than in Slack threads.

The symptom is being unable to precompile projects, it seems in some way related to CairoMakie but perhaps that's only incidental.

This is all (in my case, others seem to be experiencing this, see below) with Julia 1.9.0, initially with a juliaup install but at some point I removed that and was seeing errors with a normal /Applications install of 1.9.0. At some point I got impatient and just did a few things all at once and I'm not sure what resolved the issue for me.

All that being said I'll try to summarize the Slack discussion, for reference until consumed by the Slackhole, it's in this thread in #makie and this thread in #helpdesk.

Other Slack users (Dale Black and "@b6bd") and I have seen these errors:

Me and Dale:

ERROR: Error building `FFMPEG`: 
┌ Warning: Platform `arm64-apple-darwin22.4.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450
ERROR: LoadError: KeyError: key "unknown" not found

and (me and b6bd):

ERROR: LoadError: InitError: could not load library 
"~/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/libavdevice.58.13.100.dylib"

Here is a more complete traceback in my case:

julia> using CairoMakie
[ Info: Precompiling CairoMakie [13f3f980-e62b-5c42-98c6-ff1f3baf88f0]
ERROR: LoadError: InitError: could not load library "/Users/patrick/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/libavdevice.58.13.100.dylib"
dlopen(/Users/patrick/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/libavdevice.58.13.100.dylib, 0x0001): Library not loaded: @rpath/libssl.1.1.dylib
  Referenced from: <CB15E436-8C0A-3897-AEDE-C5935555F236> /Users/patrick/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/libavdevice.58.13.100.dylib
  Reason: tried: '/Users/patrick/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/./libssl.1.1.dylib' (no such file), '/Users/patrick/.julia/artifacts/bf37190b92ac2fc3dd5e7073ff7ec7bbfd10343f/lib/./libssl.1.1.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/julia/libssl.1.1.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/julia/../libssl.1.1.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/libssl.1.1.dylib' (no such file)

While in the same state I also saw:

julia> using MyPackage
[ Info: Precompiling MyPackage [b0850b75-0ffa-4426-82b9-d1a2be8b507a]
ERROR: LoadError: InitError: could not load library "/Users/patrick/.julia/artifacts/52d3dd71e501ee5c135601cab4e0db00ee54e620/lib/libhdf5.310.dylib"
dlopen(/Users/patrick/.julia/artifacts/52d3dd71e501ee5c135601cab4e0db00ee54e620/lib/libhdf5.310.dylib, 0x0001): Library not loaded: @rpath/libcrypto.3.dylib
  Referenced from: <B66A88F0-B162-3ADD-8073-88F9A26CA105> /Users/patrick/.julia/artifacts/52d3dd71e501ee5c135601cab4e0db00ee54e620/lib/libhdf5.310.dylib
  Reason: tried: '/Users/patrick/.julia/artifacts/52d3dd71e501ee5c135601cab4e0db00ee54e620/lib/./libcrypto.3.dylib' (no such file), '/Users/patrick/.julia/artifacts/52d3dd71e501ee5c135601cab4e0db00ee54e620/lib/./libcrypto.3.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/julia/libcrypto.3.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/julia/../libcrypto.3.dylib' (no such file), '/Users/patrick/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/lib/libcrypto.3.dylib' (no such file), '/usr/local/lib/libcrypto.3.dylib' (no such file), '/usr/lib/libcrypto.3.dylib' (no such file, not in dyld cache)

As I said I tried a bunch of things, at first trying to repro each time (though not every error message) but at a certain point I got impatient and just mashed all the buttons, so to speak (just copy-pasting my comment from Slack here):

Well, I excised juliaup and that didn't immediately change anything. However it's working again. I'm not sure what combination of these things I did since yesterday that produced that result but today I've done (in roughly this order -- but I haven't had time to be particularly meticulous about this):

• uninstalled juliaup
• manually re-installed julia (both 1.8 and 1.9) in /Applications
• removed ~/.Julia/compiled and ~/.Julia/artifacts
• removed CairoMakie from all environments in my LOAD_PATH stack (["@", "@v#.#", "@stdlib", "@patlab1.9"]) where that last one is where I install all the 'extra' packages I want available, such as Plots and Makie
• ] add CairoMakie in the @patlab1.9 environment
• changed my Julia startup script to use the default optimization level (I had been using -O3)

For now, at least, I can do using CairoMakie in my package environment (or any other).

The only other potential clue here is that I'd also been toying with the Startup.jl pattern in that @patlab1.9 environment. It's still there but AFAIU if I don't do using Startup then I think it shouldn't affect anything. But I did have CairoMakie (and GLMakie) stuff in the @compile_workload block, maybe something that PrecompileTools did got things in a weird state :shrug: .

So as it stands now unfortunately I don't know how to reproduce this on my system (willing to try with some guidance), but IIUC Dale can reproduce it; I'll point him here and he can comment if he wishes.

[gbaraldi]

Ok, so a couple of these things. The first one that mentions BinaryProvider, BinaryProvider has been unsupported for a while and doesn't work on the M1. I imagine you have an old manifest/project that's mucking it up. And I wouldn't be surprised that given that other stuff gors wrong.

Can you share the Manifest/Project files you have?

[pbouffard]

Can you share the Manifest/Project files you have?

Unfortunately no (it's for work), and as I mentioned I can't even repro at the moment. However your comment reminds me that I did notice at one point that it seemed that a very old version of FFMPEG (like. 0.2.something) was being used.

[gbaraldi]

That would do it.

[Dale-Black]

Here is a reproducer from this Project.toml file below

Simply ] activate then instantiate, and this should create the error

versioninfo()

Julia Version 1.9.0
Commit 8e630552924 (2023-05-07 11:25 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 4 on 4 virtual cores
Environment:
  JULIA_LOAD_PATH = @:@v#.#:@stdlib
  JULIA_DEPOT_PATH = /Users/daleblack/.julia:/Users/daleblack/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/local/share/julia:/Users/daleblack/.julia/juliaup/julia-1.9.0+0.aarch64.apple.darwin14/share/julia
  JULIA_REVISE_WORKER_ONLY = 1

name = "cac-dual-energy"

[deps]
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
CSVFiles = "5d742f6a-9f54-50ce-8119-2520741973ca"
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
Colors = "5ae59095-9a9b-59fe-a467-6f913c188581"
DICOM = "a26e6606-dd52-5f6a-a97f-4f611373d757"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
DrWatson = "634d3b9d-ee7a-5ddf-bec9-22491ea816e1"
GLM = "38e38edf-8417-5370-95a0-9cbb8c7f171a"
HypothesisTests = "09f84164-cd44-5f33-b23f-e6b0d136a0d5"
ImageFiltering = "6a3955dd-da59-5b1f-98d4-e7296123deb5"
ImageMorphology = "787d08f9-d448-5407-9aad-5290dd7ab264"
LsqFit = "2fda8390-95c7-5789-9bda-21331edee243"
MAT = "23992714-dd62-5051-b70f-ba57cb901cac"
MLJBase = "a7f614a8-145f-11e9-1d2a-a57a1082229d"
Noise = "81d43f40-5267-43b7-ae1c-8b967f377efa"
PlutoUI = "7f904dfe-b85e-4ff6-b463-dae2292396a8"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"

[compat]
DrWatson = "2.12.1"

  Building FFMPEG → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/9143266ba77d3313a4cf61d8333a1970e8c5d8b6/build.log`
ERROR: Error building `FFMPEG`: 
┌ Warning: Platform `arm64-apple-darwin22.4.0` is not an officially supported platform
└ @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:450
ERROR: LoadError: KeyError: key "unknown" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:484 [inlined]
  [2] parse_dl_name_version
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:511 [inlined]
  [3] detect_libgfortran_abi(libgfortran_name::String, platform::BinaryProvider.UnknownPlatform)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:555
  [4] detect_libgfortran_abi(libgfortran_name::String)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:555
  [5] detect_libgfortran_abi
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:584 [inlined]
  [6] detect_compiler_abi()
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:667
  [7] top-level scope
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:685
  [8] include(mod::Module, _path::String)
    @ Base ./Base.jl:457
  [9] include(x::String)
    @ BinaryProvider ~/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:1
 [10] top-level scope
    @ ~/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:12
 [11] include
    @ ./Base.jl:457 [inlined]
 [12] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base ./loading.jl:2010
 [13] top-level scope
    @ stdin:2
in expression starting at /Users/daleblack/.julia/packages/BinaryProvider/U2dKK/src/PlatformNames.jl:685
in expression starting at /Users/daleblack/.julia/packages/BinaryProvider/U2dKK/src/BinaryProvider.jl:1
in expression starting at stdin:2
ERROR: LoadError: Failed to precompile BinaryProvider [b99e7846-7c00-51b0-8f62-c81ae34c0232] to "/Users/daleblack/.julia/compiled/v1.9/BinaryProvider/jl_J379rJ".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2260
  [3] compilecache
    @ ./loading.jl:2127 [inlined]
  [4] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1770
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1625
  [6] macro expansion
    @ ./loading.jl:1613 [inlined]
  [7] macro expansion
    @ ./lock.jl:267 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1576
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [10] top-level scope
    @ none:5
in expression starting at /Users/daleblack/.julia/packages/FFMPEG/guN1x/deps/build.jl:1

[DilumAluthge]

@Dale-Black Can you provide the output of the following command?

julia> import Pkg; Pkg.status(; mode = Pkg.PKGMODE_MANIFEST)

[Dale-Black]

@DilumAluthge

julia> import Pkg; Pkg.status(; mode = Pkg.PKGMODE_MANIFEST)
Status `~/Library/CloudStorage/GoogleDrive-djblack@uci.edu/My Drive/dev/Molloilab/cac-dual-energy/Manifest.toml` (empty manifest)

[DilumAluthge]

Sorry, I should have clarified. First activate and instantiate your reproducer that you posted above. Then, in the same Julia session, run the command I posted.

[pbouffard]

I just had this rear its head again, in another project (a kind of OmniPackage but for the client's private packages, so again I can't share). On a whim I tried just forcing the latest FFMPEG to be explicitly included in the project (] add FFMPEG@0.4) and voila, it is resolved.

It looks like in the meantime a probably-related issue was filed over in Makie: https://github.com/MakieOrg/Makie.jl/issues/3028.