Error while compiling incremental system image with Flux

[IlyaOrson]

The script to create a sysimage in this repo throws the following error with PackageCompiler v2.0.2 while loading the project and module works fine:

[ Info: PackageCompiler: Done
⢰ [04m:07s] PackageCompiler: compiling incremental system imagefatal: error thrown and no exception handler available.
StackOverflowError()
_ZN4llvm13StringMapImpl15LookupBucketForENS_9StringRefE at C:\Users\ilyao\.julia\juliaup\julia-1.7.0-rc3+0~x64\bin\libLLVM.dll (unknown line)
_ZNK4llvm11LLVMContext11getMDKindIDENS_9StringRefE at C:\Users\ilyao\.julia\juliaup\julia-1.7.0-rc3+0~x64\bin\libLLVM.dll (unknown line)
_ZNK4llvm5Value11getMetadataENS_9StringRefE at C:\Users\ilyao\.julia\juliaup\julia-1.7.0-rc3+0~x64\bin\libLLVM.dll (unknown line)
✖ [04m:08s] PackageCompiler: compiling incremental system image
ERROR: LoadError: failed process: Process(`'C:\Users\ilyao\.julia\juliaup\julia-1.7.0-rc3+0~x64\bin\julia.exe' --color=yes --startup-file=no --cpu-target=native -O3 '--sysimage=C:\Users\ilyao\.julia\juliaup\julia-1.7.0-rc3+0~x64\lib\julia\sys.dll' '--project=D:\ilyao\Documents\research\neural_ode\control_neuralode' '--output-o=C:\Users\ilyao\AppData\Local\Temp\jl_ujXfJKYsNX.o' 'C:\Users\ilyao\AppData\Local\Temp\jl_6Je3AZ283E'`, ProcessExited(3221225477)) [3221225477]

Stacktrace:
 [1] pipeline_error
   @ .\process.jl:531 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base .\process.jl:446
 [3] run
   @ .\process.jl:444 [inlined]
 [4] #14
   @ C:\Users\ilyao\.julia\packages\PackageCompiler\R3n2H\ext\TerminalSpinners.jl:157 [inlined]
 [5] spin(f::PackageCompiler.var"#14#15"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
   @ PackageCompiler.TerminalSpinners C:\Users\ilyao\.julia\packages\PackageCompiler\R3n2H\ext\TerminalSpinners.jl:164
 [6] macro expansion
   @ C:\Users\ilyao\.julia\packages\PackageCompiler\R3n2H\ext\TerminalSpinners.jl:157 [inlined]
 [7] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd)
   @ PackageCompiler C:\Users\ilyao\.julia\packages\PackageCompiler\R3n2H\src\PackageCompiler.jl:352
 [8] create_sysimage(packages::Vector{Symbol}; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing,
 version::Nothing, soname::Nothing, compat_level::String)
   @ PackageCompiler C:\Users\ilyao\.julia\packages\PackageCompiler\R3n2H\src\PackageCompiler.jl:502
 [9] top-level scope
   @ D:\ilyao\Documents\research\neural_ode\control_neuralode\create_sysimage.jl:11
in expression starting at D:\ilyao\Documents\research\neural_ode\control_neuralode\create_sysimage.jl:11

Not sure if this warning is related to the issue, but it appears when loading the project:

WARNING: Method definition (::Type{DiffEqFlux.DeterministicCNF{M, P, RE, D, T, A, K} where K where A where T where D where RE where P where M})(M, P, RE, D, T, A, K) where {M, P, RE, D, T, A, K} in module DiffEqFlux at deprecated.jl:70 overwritten at C:\Users\ilyao\.julia\packages\DiffEqFlux\jpIWG\src\ffjord.jl:41.
  ** incremental compilation may be fatally broken for this module **

My version info:

julia> versioninfo()
Julia Version 1.7.0-rc3
Commit 3348de4ea6 (2021-11-15 08:22 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 7 4800H with Radeon Graphics
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)

[IlyaOrson]

The problem is Flux v0.12.8 on julia v1.7.

I can reproduce by running create_sysimage([:Flux]; sysimage_path="flux.so", precompile_execution_file="Foo.jl") with the following precompile file:

module Foo
using Flux
end

I think this is a duplicate of #643

[toollu]

Yeah I have similar issues/error when trying to create_app with flux. It however works on V1.7.6. The app is not relocatable however then due to this line. When I don't include Data.jl in Flux it however works. Maybe this is the cause for the incremental compiling failing on versions >V1.7.6 too?

[mashu]

Same problem with Julia 1.7.1 and Julia 1.6.5

pkg> st
    Project tool v0.1.0
     Status `~/tool/Project.toml`
  [052768ef] CUDA v3.6.2
  [587475ba] Flux v0.12.8
  [9b87118b] PackageCompiler v2.0.4
julia> create_app("./","tool")
PackageCompiler: bundled artifacts:
  ├── LLVMExtra_jll - 5.370 MiB
  └── OpenSpecFun_jll - 577.252 KiB
  Total artifact file size: 5.934 MiB
✔ [02m:24s] PackageCompiler: compiling base system image (incremental=false)
Precompiling project...
  73 dependencies successfully precompiled in 302 seconds
⠙ [00m:39s] PackageCompiler: compiling incremental system imagejulia: /buildworker/worker/package_linux64/build/usr/include/llvm/Support/Casting.h:269: typename llvm::cast_retty<X, Y*>::ret_type llvm::cast(Y*) [with X = llvm::GlobalVariable; Y = llvm::GlobalValue; typename llvm::cast_retty<X, Y*>::ret_type = llvm::GlobalVariable*]: Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

signal (6): Aborted
in expression starting at none:0
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f8b77daa41e)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
cast<llvm::GlobalVariable, llvm::GlobalValue> at /buildworker/worker/package_linux64/build/usr/include/llvm/Support/Casting.h:269 [inlined]
operator() at /buildworker/worker/package_linux64/build/src/llvm-multiversioning.cpp:711
fix_gv_uses at /buildworker/worker/package_linux64/build/src/llvm-multiversioning.cpp:726 [inlined]
runOnModule at /buildworker/worker/package_linux64/build/src/llvm-multiversioning.cpp:1053
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia-1.7.1/bin/../lib/julia/libLLVM-12jl.so (unknown line)
operator() at /buildworker/worker/package_linux64/build/src/aotcompile.cpp:556 [inlined]
jl_dump_native at /buildworker/worker/package_linux64/build/src/aotcompile.cpp:567
jl_write_compiler_output at /buildworker/worker/package_linux64/build/src/precompile.c:94
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:211
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /opt/julia-1.7.1/bin/julia (unknown line)
Allocations: 75582852 (Pool: 75557635; Big: 25217); GC: 59
✖ [00m:40s] PackageCompiler: compiling incremental system image
ERROR: failed process: Process(`/opt/julia-1.7.1/bin/julia --color=yes --startup-file=no '--cpu-target=generic;sandybridge,-xsaveopt,clone_all;haswell,-rdrnd,base(1)' -O3 --sysimage=/tmp/jl_r3Kswn/sys.ji --project=/home/mateusz/tool --output-o=/tmp/jl_lZULta.o /tmp/jl_561Amj`, ProcessSignaled(6)) [0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:531 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:446
  [3] run
    @ ./process.jl:444 [inlined]
  [4] #14
    @ ~/.julia/packages/PackageCompiler/tdgyo/ext/TerminalSpinners.jl:157 [inlined]
  [5] spin(f::PackageCompiler.var"#14#15"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners ~/.julia/packages/PackageCompiler/tdgyo/ext/TerminalSpinners.jl:164
  [6] macro expansion
    @ ~/.julia/packages/PackageCompiler/tdgyo/ext/TerminalSpinners.jl:157 [inlined]
  [7] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/tdgyo/src/PackageCompiler.jl:356
  [8] create_sysimage(packages::Vector{String}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/tdgyo/src/PackageCompiler.jl:507
  [9] create_app(package_dir::String, app_dir::String; executables::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, force::Bool, c_driver_program::String, cpu_target::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/tdgyo/src/PackageCompiler.jl:718
 [10] create_app(package_dir::String, app_dir::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/tdgyo/src/PackageCompiler.jl:689
 [11] top-level scope
    @ REPL[8]:1

[KristofferC]

Same problem with Julia 1.7.1 and Julia 1.6.5

I think this is different, for apps you need to remove any @ccallable that you might have left from using the earlier PackageCompiler.

[KristofferC]

Made an issue upstream: https://github.com/JuliaLang/julia/issues/43665.

[KristofferC]

Should be fixed on the upcoming Julia 1.7.2

[KristofferC]

This should have been fixed in 1.7.2. Please comment if you still see this on 1.7.2.

[mashu]

I reported this issue here https://github.com/JuliaGPU/CUDA.jl/issues/1365#issuecomment-1035056509 As it seems to be related not to Flux but to it's dependency CUDA.jl

The problem was reproducible on all Julia versions I tested 1.6.5, 1.7.1 and 1.7.2.