search

JuliaPackaging / Yggdrasil

Collection of builder repositories for BinaryBuilder.jl
https://binarybuilder.org
Other
308 stars 553 forks source link

CUDNN_jll is breaking artifacts and segfaulting #6626

Closed rafaqz closed 1 year ago

rafaqz commented 1 year ago

I keep hitting this with anything to do with Pkg, when it runs collect_artifacts. It happens from many commands but always segaults in CUDNN_jll builds

(@v1.9) pkg> add --preserve=all TerminalPager
   Resolving package versions...
error: <inline asm>:1:2: invalid character in input
        �"j

[20771] signal (11.128): Segmentation fault
in expression starting at none:0
unknown function (ip: 0x7fe56ab4b3ac)
exit at /usr/lib/libc.so.6 (unknown line)
_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown li
ne)
_ZNSt17_Function_handlerIFvRKN4llvm12SMDiagnosticEbRKNS0_9SourceMgrERSt6vectorIPKNS0_6MDNodeESaISA_EEEZNS0_28Mac
hineModuleInfoWrapperPass16doInitializationERNS0_6ModuleEEUlS3_bS6_SD_E_E9_M_invokeERKSt9_Any_dataS3_ObS6_SD_ at
 /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm9MCContext8diagnoseERKNS_12SMDiagnosticE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser11DiagHandlerERKN4llvm12SMDiagnosticEPv at /opt/julia/bin/../lib/julia/libLLVM-14jl.s
o (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageERNS_11raw_ostreamENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRange
EEENS8_INS_7SMFixItEEEb at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm9SourceMgr12PrintMessageENS_5SMLocENS0_8DiagKindERKNS_5TwineENS_8ArrayRefINS_7SMRangeEEENS6_INS_7SMFixI
tEEEb at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN12_GLOBAL__N_19AsmParser3RunEbb at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmENS_9StringRefERKNS_15MCSubtargetInfoERKNS_15MCTargetOptionsEPKNS_6MDNodeENS
_9InlineAsm10AsmDialectE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZNK4llvm10AsmPrinter13emitInlineAsmEPKNS_12MachineInstrE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknow
n line)
_ZN4llvm10AsmPrinter16emitFunctionBodyEv at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM-14jl
.so (unknown line)
_ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unkn
own line)
_ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown li
ne)
_ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
operator() at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1206
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0
_16ThreadSafeModuleE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EEN
S0_16ThreadSafeModuleE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
emit at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:631
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt
14default_deleteIS3_EE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEv
PvRS7_ at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown
 line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteI
S3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_
15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /opt/julia/bin/.
./lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS
3_EE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_delet
eIS3_EENS_5ErrorE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFl
agsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15Symb
olStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionI
FvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (un
known line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EERKNS
0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15Symbol
StringPtrENS_12DenseMapInfoISK_vEEEENSL_IS5_vEENS_6detail12DenseMapPairIS5_SN_EEEEEE at /opt/julia/bin/../lib/ju
lia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_
15SymbolStringPtrENS0_11SymbolStateE at /opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at 
/opt/julia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /opt/jul
ia/bin/../lib/julia/libLLVM-14jl.so (unknown line)
addModule at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1420
jl_add_to_ee at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:1815
_jl_compile_codeinst at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:241
jl_generate_fptr_impl at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/jitlayers.cpp:460
jl_compile_method_internal at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2348 [inlin
ed]
jl_compile_method_internal at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2237
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2750 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
print at ./strings/io.jl:46
show at ./stream.jl:263 [inlined]
print at ./strings/io.jl:35
jfptr_print_33427.clone_1 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
print_to_string at ./strings/io.jl:144
string at ./strings/io.jl:185 [inlined]
isopen at ./stream.jl:381 [inlined]
check_open at ./stream.jl:387 [inlined]
displaysize at ./stream.jl:569
jfptr_displaysize_53371.clone_1 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
#handle_message#3 at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/L
ogging/src/ConsoleLogger.jl:124
handle_message at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdlib/v1.9/Logg
ing/src/ConsoleLogger.jl:106
unknown function (ip: 0x7fe5501673d8)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_f__call_latest at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:774
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
#invokelatest#2 at ./essentials.jl:818
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/builtins.c:730
invokelatest at ./essentials.jl:813
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
macro expansion at ./logging.jl:365 [inlined]
profile_printing_listener at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/usr/share/julia/stdli
b/v1.9/Profile/src/Profile.jl:49
#3 at ./threadingconstructs.jl:341
jfptr_YY.3_59756.clone_1 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
start_task at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-9/src/task.c:1092
Allocations: 2995 (Pool: 2983; Big: 12); GC: 0
ERROR: failed process: Process(`/opt/julia/bin/julia -Cnative -J/opt/julia/lib/julia/sys.so --check-bounds=yes -
g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/raf/.julia/environm
ents/v1.9/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/raf/.julia", "/opt/julia/local/share/jul
ia", "/opt/julia/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/raf/.julia/packages/CUDNN_jll/npufe/.pkg")
include("/home/raf/.julia/packages/CUDNN_jll/npufe/.pkg/select_artifacts.jl")
' --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.9.0`, ProcessSignaled(11)) [
0]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:721
  [4] collect_artifacts
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:707 [inlined]
  [5] download_artifacts(env::Pkg.Types.EnvCache; platform::Base.BinaryPlatforms.Platform, julia_version::Versio
nNumber, verbose::Bool, io::Base.TTY)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:753
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Ty
pes.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1390
  [7] add
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372 [inlined]
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platfo
rm::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{
Base.TTY}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
  [9] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Pkg.Types.PreserveLevel,
 Tuple{Symbol}, NamedTuple{(:preserve,), Tuple{Pkg.Types.PreserveLevel}}})
    @ Pkg.API /opt/julia/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [10] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /opt/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:412
 [11] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /opt/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:390
 [12] do_cmd
    @ /opt/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [13] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuf
fer, ok::Bool)
    @ Pkg.REPLMode /opt/julia/share/julia/stdlib/v1.9/Pkg/src/REPLMode/REPLMode.jl:557
 [14] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [15] invokelatest
    @ ./essentials.jl:813 [inlined]
 [16] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIS
tate)
    @ REPL.LineEdit /opt/julia/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [17] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /opt/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1298
 [18] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514
julia> versioninfo()
Julia Version 1.9.0-rc2
Commit 72aec423c2a (2023-04-01 10:41 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 12 × Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 6 on 12 virtual cores
Environment:
  JULIA_NUM_THREADS = 5
  JULIA_EDITOR = lvim

And the entry for CUDNN_jll in my manifest:

[[deps.CUDNN_jll]]
deps = ["Artifacts", "CUDA_Runtime_jll", "JLLWrappers", "LazyArtifacts", "Libdl", "TOML"]
git-tree-sha1 = "2918fbffb50e3b7a0b9127617587afa76d4276e8"
uuid = "62b44479-cb7b-5706-934f-f13b2eb2e645"
version = "8.8.1+0"
giordano commented 1 year ago

@maleadt

maleadt commented 1 year ago

I'm aware, but unsure what to do. It seems like a Julia bug triggered by CUDNN's artifact selection, but that doesn't involve doing anything weird with LLVM (only loading CUDA drivers and libraries, which has worked out fine with other CUDA JLLs). Without a consistent reproducer, this is almost impossible to debug.

rafaqz commented 1 year ago

I can semi-consistently reproduce this in my system. Its seems to be when I add anything when some packages are already imported in the session.

It doesnt happen in a fresh session, and it doesnt happen after I delete the compiled folder.

I will try and catch more of the logic of it, but doing it intentionally requires delering the compiled folder so the iteration time is very slow.

maleadt commented 1 year ago

Ideally we'd capture this in rr. Normally that just requires running Julia with --bug-report, but in the case of a precompilation process that doesn't really work...

rafaqz commented 1 year ago

Because its happening in a separate julia process?

Can we just add --bug-report to the call to the julia process in Pkg.Operations.collect_artifacts? (I don't really use rr so just a guess)

A very consistent failure I'm getting is on resolve when I already have the packages loaded in the session.

maleadt commented 1 year ago

Can we just add --bug-report to the call to the julia process in Pkg.Operations.collect_artifacts? (I don't really use rr so just a guess)

We could, but --bug-report ends with an interactive part where it prompts you to do something in the browser in order to upload a report. Although I guess you could have it run with --bug-report=rr-local, which disables the upload and just puts the trace in ~/.rr. One potential issue with that, is that --bug-report will attempt to install BugReporting.jl, and thus perform Pkg operations and do some precompilation, which may conflict with the environment that the precompilation process runs in (and/or cause recursion).

It may be easier to install rr locally and just prefix the precompilation spawn with rr record and do some of the BugReport.jl-postprocessing manually afterwards.

rafaqz commented 1 year ago

Ok probably manually then, not sure I have the time or experience to do this currently.

What I'm finding strange is that resolve in a fresh session never seems to break, but nearly always breaks when packages are loaded.

So why does the main process interact with the artifacts Julia process in any way? I didn't think they would be sharing anything that's in memory?

maleadt commented 1 year ago

Ok probably manually then, not sure I have the time or experience to do this currently.

Alternatively, if you can come up with something that reproduces deterministically (e.g., starting from a fresh depot by setting JULIA_DEPOTPATH=$(mktemp -d)) I can take a look at trying to shoehorn rr into it.

rafaqz commented 1 year ago

I just got the segfault in a fresh session so that theory is gone too. Will see if I can find some time to make something deterministic.

Moelf commented 1 year ago 
(WVZAnalysis) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
   Installed XGBoost_jll ─ v1.7.5+0
error: <inline asm>:1:2: invalid character in input
        4�

[2837042] signal (11.1): Segmentation fault
in expression starting at none:0
__run_exit_handlers at /lib64/libc.so.6 (unknown line)
exit at /lib64/libc.so.6 (unknown line)
*** Error in `/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia': corrupted double-linked list: 0x0000000001ae10c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x80a4f)[0x7f3c67c80a4f]
/lib64/libc.so.6(+0x82135)[0x7f3c67c82135]
/lib64/libc.so.6(__libc_calloc+0xb4)[0x7f3c67c86214]
/home/jiling/.julia/artifacts/ebadc1bf983003ca3f714f062af4451365761171/lib/libcublasLt.so.11(+0x5a224c3)[0x7f39ebe3e4c3]
/home/jiling/.julia/artifacts/ebadc1bf983003ca3f714f062af4451365761171/lib/libcublasLt.so.11(+0x5a23b78)[0x7f39ebe3fb78]
/lib64/libc.so.6(+0x39ce9)[0x7f3c67c39ce9]
/lib64/libc.so.6(+0x39d37)[0x7f3c67c39d37]
/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia(_start+0x0)[0x401070]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f3c67c22555]
/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia[0x401099]
======= Memory map: ========
00400000-00401000 r--p 00000000 00:69 3713014                            /cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia
00401000-00402000 r-xp 00001000 00:69 3713014                            /cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia
00402000-00403000 r--p 00002000 00:69 3713014                            /cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia
00403000-00404000 r--p 00002000 00:69 3713014                            /cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia
00404000-00405000 rw-p 00003000 00:69 3713014                            /cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia
011e0000-020a3000 rw-p 00000000 00:00 0                                  [heap]
7f398c000000-7f398c021000 rw-p 00000000 00:00 0
7f398c021000-7f3990000000 ---p 00000000 00:00 0
7f3990a00000-7f3_ZN4llvm11LLVMContext8diagnoseERKNS_14DiagnosticInfoE at /cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Thu/x86_64-centos7-gcc11-opt/lib/julia/libLLVM-14jl.so (unknown line)

[2837042] signal (6.-6): Aborted
in expression starting at none:0
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
__libc_message at /lib64/libc.so.6 (unknown line)
malloc_consolidate at /lib64/libc.so.6 (unknown line)
_int_malloc at /lib64/libc.so.6 (unknown line)
__libc_calloc at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x7f39ebe3e4c2)
unknown function (ip: 0x7f39ebe3fb77)
__run_exit_handlers at /lib64/libc.so.6 (unknown line)
exit at /lib64/libc.so.6 (unknown line)
main at /cache/build/default-amdci4-0/julialang/julia-release-1-dot-9/cli/loader_exe.c:62
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 2997 (Pool: 2984; Big: 13); GC: 0
ERROR: failed process: Process(`/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/bin/julia -Cnative -J/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/home/jiling/.julia/dev/WVZAnalysis/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/jiling/.julia", "/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/local/share/julia", "/cvmfs/sft-nightlies.cern.ch/lcg/latest/julia/1.9.0-30f63/x86_64-centos7-gcc11-opt/share/julia", "/cvmfs/sft-nightlies.cern.ch/lcg/views/dev4/Thu/x86_64-centos7-gcc11-opt/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

this is now hitting XGBoost_jll starting 1.7.4 and 1.7.5

rafaqz commented 1 year ago

It stopped happening for me after updating my julia version.

maleadt commented 1 year ago

I haven't encountered it recently either.