Failing to precompile Flux and Zygote

[yuvalwas]

Hello, I failed to precompile Zygore on my Institute's HPC. Would appreciate any help.

Stacktrace

Precompiling project...  ✗ Zygote  ✗ Flux 0 dependencies successfully precompiled in 112 seconds (337 already precompiled)

ERROR: LoadError: The following 1 direct dependency failed to precompile:

Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]

Failed to precompile Flux [587475ba-b771-5e3f-ad9e-33799f191a9c] to /home/labs/tsodyks/yuvalw/.julia/compiled/v1.7/Flux/jl_6Q1dTi. ERROR: LoadError: MethodError: no method matching iterate(::Core.CodeInfo) Closest candidates are: iterate(!Matched::Union{LinRange, StepRangeLen}) at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/range.jl:826 iterate(!Matched::Union{LinRange, StepRangeLen}, !Matched::Integer) at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/range.jl:826 iterate(!Matched::T) where T<:Union{Base.KeySet{<:Any, <:Dict}, Base.ValueIterator{<:Dict}} at /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/base/dict.jl:695 ... Stacktrace: [1] indexed_iterate(I::Core.CodeInfo, i::Int64) @ Base ./tuple.jl:92 [2] pullback(::Function, ::Zygote.Context{false}, ::Int64, ::Vararg{Int64}) @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:44 [3] pullback(::Function, ::Int64, ::Int64) @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:42 [4] gradient(::Function, ::Int64, ::Vararg{Int64}) @ Zygote ~/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:96 [5] top-level scope @ ~/.julia/packages/Zygote/4SSHS/src/precompile.jl:17 [6] include(mod::Module, _path::String) @ Base ./Base.jl:418 [7] include @ ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1 [inlined] [8] macro expansion @ ~/.julia/packages/Requires/Z8rfN/src/Requires.jl:40 [inlined] [9] precompile() @ Zygote ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:63 [10] macro expansion @ ~/.julia/packages/PrecompileTools/kmH5L/src/workloads.jl:85 [inlined] [11] top-level scope @ ~/.julia/packages/Zygote/4SSHS/src/Zygote.jl:84 [12] include @ ./Base.jl:418 [inlined] [13] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String) @ Base ./loading.jl:1318 [14] top-level scope @ none:1 [15] eval @ ./boot.jl:373 [inlined] [16] eval(x::Expr) @ Base.MainInclude ./client.jl:453 [17] top-level scope @ none:1 in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Zygote/4SSHS/src/precompile.jl:17 in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1 ERROR: LoadError: Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to /home/labs/tsodyks/yuvalw/.julia/compiled/v1.7/Zygote/jl_vTqQFV. Stacktrace: [1] error(s::String) @ Base ./error.jl:33 [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool) @ Base ./loading.jl:1466 [3] compilecache(pkg::Base.PkgId, path::String) @ Base ./loading.jl:1410 [4] _require(pkg::Base.PkgId) @ Base ./loading.jl:1120 [5] require(uuidkey::Base.PkgId) @ Base ./loading.jl:1013 [6] require(into::Module, mod::Symbol) @ Base ./loading.jl:997 [7] include @ ./Base.jl:418 [inlined] [8] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing) @ Base ./loading.jl:1318 [9] top-level scope @ none:1 [10] eval @ ./boot.jl:373 [inlined] [11] eval(x::Expr) @ Base.MainInclude ./client.jl:453 [12] top-level scope @ none:1 in expression starting at /home/labs/tsodyks/yuvalw/.julia/packages/Flux/n3cOc/src/Flux.jl:1 Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68 [2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ Pkg.API /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1362 [3] precompile @ /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1013 [inlined] [4] #precompile#220 @ /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1011 [inlined] [5] precompile() @ Pkg.API /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/share/julia/stdlib/v1.7/Pkg/src/API.jl:1011 [6] top-level scope @ ~/clusterless/wexac_utils/env_setup.jl:10 in expression starting at /home/labs/tsodyks/yuvalw/clusterless/wexac_utils/env_setup.jl:5

# Julia version

julia> versioninfo() Julia Version 1.7.1 Commit ac5cc99908 (2021-12-22 19:35 UTC) Platform Info: OS: Linux (x86_64-pc-linux-gnu) CPU: Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-12.0.1 (ORCJIT, icelake-server) Environment: EBVERSIONJULIA = 1.7.1 EBROOTJULIA = /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64 EBDEVELJULIA = /apps/easybd/easybuild/software/Julia/1.7.1-linux-x86_64/easybuild/Julia-1.7.1-linux-x86_64-easybuild-devel

# Zygote and Flux versions
Zygote v0.6.64
Flux v0.13.17

[yuvalwas]

I see that this is similar to #1449. Do you think working with the latest Julia version would solve this?

[ToucheSir]

That would be ideal. It's possible some 1.7 and/or 1.8 versions may have received the same backport fix that was mentioned in #1449, but 1.7.1 likely wouldn't have because of how old it is.

[voroskoi]

Hi,

I have the same issue with julia-1.9.3. If I try to precompile just Zygote that also fails without any error message, it just shows a red X.

Here is the error message:

julia> Pkg.precompile("Zygote"; strict=true)
Precompiling Zygote
  ✗ Zygote
  0 dependencies successfully precompiled in 16 seconds. 56 already precompiled.

ERROR: The following 1 dependency failed to precompile:

Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to "/home/voroskoi/.julia/compiled/v1.9/Zygote/jl_rZ7rSE".
Illegal inttoptr
          %magicptr = ptrtoint {} addrspace(10)* %242 to i64, !dbg !26

[82350] signal (6.-6): Aborted
in expression starting at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/precompile.jl:13
unknown function (ip: 0x7fd1d868e83c)
raise at /usr/bin/../lib/libc.so.6 (unknown line)
abort at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x7fd1d7a7dff7)
unknown function (ip: 0x7fd1d7ba8d88)
_ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba7db8)
_ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba85f8)
_ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7ba6081)
unknown function (ip: 0x7fd1d7b13f0f)
unknown function (ip: 0x7fd1d7b148ac)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7b0baea)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d305ac6b)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d307cf7e)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7fd1d7b0f059)
unknown function (ip: 0x7fd1d7b10a69)
unknown function (ip: 0x7fd1d7b11882)
jl_generate_fptr_impl at /usr/bin/../lib/julia/libjulia-codegen.so.1 (unknown line)
jl_compile_method_internal at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
ijl_apply_generic at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
#166 at ./none:0
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:840
collect_to_with_first! at ./array.jl:818
unknown function (ip: 0x7fd196f939f5)
collect at ./array.jl:792
unknown function (ip: 0x7fd196f8e716)
prune! at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/passes/passes.jl:181
|> at ./operators.jl:907 [inlined]
#IR#15 at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:151 [inlined]
IR at /home/voroskoi/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:146 [inlined]
code_ir at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/compiler/interface.jl:428
unknown function (ip: 0x7fd196f652e8)
unknown function (ip: 0x7fd1d7c64e2c)
unknown function (ip: 0x7fd1d7c6460a)
unknown function (ip: 0x7fd1d7c6572b)
unknown function (ip: 0x7fd1d7c6660e)
unknown function (ip: 0x7fd1d7c82942)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1be86d3b8)
unknown function (ip: 0x7fd1be86d4f2)
unknown function (ip: 0x7fd1be86d50f)
jl_f__call_latest at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
include at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:1 [inlined]
macro expansion at /home/voroskoi/.julia/packages/Requires/Z8rfN/src/Requires.jl:40 [inlined]
precompile at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:63
macro expansion at /home/voroskoi/.julia/packages/PrecompileTools/kmH5L/src/workloads.jl:78 [inlined]
top-level scope at /home/voroskoi/.julia/packages/Zygote/4SSHS/src/Zygote.jl:84
unknown function (ip: 0x7fd1d7c82cfc)
unknown function (ip: 0x7fd1d7c83011)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1be86d3b8)
unknown function (ip: 0x7fd1be6ab891)
unknown function (ip: 0x7fd1be6abab2)
unknown function (ip: 0x7fd1d7c64e2c)
unknown function (ip: 0x7fd1d7c6460a)
unknown function (ip: 0x7fd1d7c6572b)
unknown function (ip: 0x7fd1d7c6660e)
unknown function (ip: 0x7fd1d7c82942)
unknown function (ip: 0x7fd1d7c83359)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7fd1bebbd6fe)
unknown function (ip: 0x7fd1bec59fdc)
unknown function (ip: 0x7fd1bec5a49f)
unknown function (ip: 0x7fd1bec5a5a8)
unknown function (ip: 0x7fd1d7cb0ff0)
jl_repl_entrypoint at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
main at /usr/bin/julia (unknown line)
unknown function (ip: 0x7fd1d8627ccf)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
_start at /usr/bin/julia (unknown line)
Allocations: 16079812 (Pool: 16069899; Big: 9913); GC: 24
Stacktrace:
 [1] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:strict,), Tuple{Bool}}})
   @ Pkg.API /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [2] #precompile#115
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
 [3] precompile
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
 [4] #precompile#114
   @ /usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [5] top-level scope
   @ REPL[2]:1

julia>

[ToucheSir]

That sounds more like https://github.com/FluxML/Zygote.jl/issues/1455 since you're using a recent version of Julia. I'd try the steps in https://github.com/FluxML/Zygote.jl/issues/1455#issuecomment-1720448912 and open a new thread if they don't work.

[voroskoi]

I have opened #1457 after trying to build in a clean environment.

[yuvalwas]

Updating to 1.9.3. more or less worked for me. (Some other packages then failed 😞 )