search

JuliaGPU / GPUCompiler.jl

Reusable compiler infrastructure for Julia GPU backends.
Other
150 stars 44 forks source link

`illegal hardware instruction` via precompiling CPU and CUDA KA kernels #580

Open lcw opened 1 month ago

lcw commented 1 month ago

I tried to add precompilation to my package (via PrecompileTools in https://github.com/HorribleSanity/Raven.jl/commit/52e0cc1820176021890e3c5c10414311b4593ed4) and am now generating illegal hardware instruction errors. I have not been able to minimize it but @vchuravy asked me to submit an issue so here it is. I have been able to reproduce the issue on two different machines via

rm -rf ~/.julia
git clone -b lcw/precompile_tools git@github.com:HorribleSanity/Raven.jl
cd Raven.jl
julia -e "using Pkg; Pkg.add(\"CUDA\")"
julia --project -e "using Pkg; Pkg.instantiate()"
julia --project -e "using CUDA, Raven"
julia --project -e "using CUDA, Raven; generate(GridManager(LobattoCell{Float32,Array}(3,4,5),brick(3,3,2)))"

The stack trace that is produced is

Internal error: encountered unexpected error in runtime:
AssertionError(msg="malformed :new expression")
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2405
reprocess_instruction! at ./compiler/ssair/irinterp.jl:126
#_ir_abstract_constant_propagation#511 at ./compiler/ssair/irinterp.jl:251
_ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:206
ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:384
semi_concrete_eval_call at ./compiler/abstractinterpretation.jl:1168
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:812
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:788
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:103
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45276.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2480 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2886 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
top-level scope at none:1
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Internal error: encountered unexpected error in runtime:
AssertionError(msg="malformed :new expression")
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2405
reprocess_instruction! at ./compiler/ssair/irinterp.jl:126
#_ir_abstract_constant_propagation#511 at ./compiler/ssair/irinterp.jl:251
_ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:206
ir_abstract_constant_propagation at ./compiler/ssair/irinterp.jl:384
semi_concrete_eval_call at ./compiler/abstractinterpretation.jl:1168
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:812
abstract_call_method_with_const_args at ./compiler/abstractinterpretation.jl:788
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:103
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45276.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jitlayers.cpp:502
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2480 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2886 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
__run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:82
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
top-level scope at none:1
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
warning: /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/extras/loopinfo.jl:26:0: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering
Unreachable reached at 0x7f114e849c5f

[666880] signal (4.2): Illegal instruction
in expression starting at none:1
NamedTuple at ./boot.jl:622
unknown function (ip: 0x7f114e849c99)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
macro expansion at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/extras/loopinfo.jl:26 [inlined]
macro expansion at /tmp/lucas/Raven.jl/src/lobattocells.jl:2729 [inlined]
cpu_hexvolumebrickmetrics! at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/macros.jl:287 [inlined]
cpu_hexvolumebrickmetrics! at ./none:0
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
__thread_run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:115
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
__run at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:82
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
#_#16 at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:44
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/builtins.c:768
Kernel at /home/lwilcox/.julia/packages/KernelAbstractions/zPAn3/src/cpu.jl:37
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
materializemetrics at /tmp/lucas/Raven.jl/src/lobattocells.jl:2849
unknown function (ip: 0x7f114e845a22)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:422
generate at /tmp/lucas/Raven.jl/src/gridmanager.jl:127
unknown function (ip: 0x7f114e840015)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
exec_options at ./client.jl:291
_start at ./client.jl:552
jfptr__start_82738.1 at /home/lwilcox/local/julia/1.10.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:2894 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/gf.c:3076
jl_apply at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-1/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at julia (unknown line)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 6252595 (Pool: 6247471; Big: 5124); GC: 8

The julia version is

Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  uname: Linux 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64
  CPU: AMD EPYC 7443P 24-Core Processor: 
                 speed         user         nice          sys         idle          irq
       #1-48  3478 MHz     182186 s      12777 s     469537 s  2330423177 s      41199 s
  Memory: 251.2397003173828 GB (252315.3515625 MB free)
  Uptime: 4.8571565e6 sec
  Load Avg:  0.44  2.3  2.71
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 48 virtual cores)
Environment:
  JULIA_CPU_TARGET = haswell,clone_all;broadwell,base(0);skylake-avx512,base(0);znver2,base(0);znver3,base(0)
  LD_LIBRARY_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib
  LD_LIBRARY_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib:1
  ACLOCAL_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/share/aclocal
  ACLOCAL_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/share/aclocal:1
  CMAKE_PREFIX_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/
  CMAKE_PREFIX_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/:1
  HOME = /home/lwilcox
  LIBRARY_PATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib
  LIBRARY_PATH_modshare = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/lib:1
  MANPATH = /home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/share/man:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/man:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/share/man::::/home/lwilcox/.fzf/man
  MANPATH_modshare = :1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/man:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/share/man:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/share/man:1
  MODULEPATH = /etc/scl/modulefiles:/usr/share/Modules/modulefiles:/share/modules/base
  MODULEPATH_modshare = /home/lwilcox/spack/share/spack/modules/linux-centos7-x86_64:1:/etc/scl/modulefiles:1:/share/modules/base:2:/usr/share/Modules/modulefiles:1
  MODULESHOME = /usr/share/Modules
  PATH = /home/lwilcox/.local/bin:/home/lwilcox/local/bin:/usr/share/Modules/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/bin:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/environment-modules-4.6.1-zdvbiw6n4iwdghhmstuqqewwl4hjdfsg/bin:/home/lwilcox/.local/bin:/home/lwilcox/local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/lwilcox/.fzf/bin:/home/lwilcox/bin:/home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin/bin
  PATH_modshare = /home/lwilcox/.local/bin:1:/usr/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/cmake-3.19.2-j5ueho7ufuao3ifxfyuxzed2ujrb3i6p/bin:1:/home/lwilcox/local/bin:1:/usr/local/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tig-2.2.2-ftzfa46wq5l6sklevjrv3lin7gwvbnch/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/git-2.29.0-qbsprtvnye6gpbamjvqsozdjjmkydrhr/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/environment-modules-4.6.1-zdvbiw6n4iwdghhmstuqqewwl4hjdfsg/bin:1:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/tmux-3.1b-qfqajvvt23flztkmtjzrx4wdc3hry5rv/bin:1:/usr/sbin:1:/home/lwilcox/.fzf/bin:1:/usr/local/sbin:1
  TERM = tmux-256color
  JAVA_HOME = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-2.el8.x86_64/jre
  FPATH = /home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin/completions:/usr/share/Modules/init/zsh-functions:/usr/local/share/zsh/site-functions:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/zsh/site-functions:/home/lwilcox/spack/opt/spack/linux-centos7-x86_64/gcc-4.8.5/zsh-5.8-zlms4pldvxvpeybpnk44z5fclyqrdvf6/share/zsh/5.8/functions:/home/lwilcox/.config/zsh/plugins/zsh-defer:/home/lwilcox/.config/zsh/plugins/zsh-completions:/home/lwilcox/.config/zsh/plugins/zsh-history-substring-search:/home/lwilcox/.config/zsh/plugins/zsh-vi-mode:/home/lwilcox/.config/zsh/plugins/fzf-zsh-plugin:/home/lwilcox/.config/zsh/plugins/fzf-tab:/home/lwilcox/.config/zsh/plugins/zsh-autosuggestions:/home/lwilcox/.config/zsh/plugins/fast-syntax-highlighting:/home/lwilcox/.config/zsh/plugins/zsh-completions/src:/home/lwilcox/.config/zsh/plugins/fzf-tab/lib

The global environment

Status `~/.julia/environments/v1.10/Manifest.toml`                                                                                                                                                                  
  [621f4979] AbstractFFTs v1.5.0                                                                                                                                                                                    
  [79e6a3ab] Adapt v4.0.4                                                                                                                                                                                           
  [a9b6321e] Atomix v0.1.0                                                                                                                                                                                          
  [ab4f0b2a] BFloat16s v0.5.0                                                                                                                                                                                       
  [fa961155] CEnum v0.5.0                                                                                                                                                                                           
  [052768ef] CUDA v5.3.3                                                                                                                                                                                            
⌅ [1af6417a] CUDA_Runtime_Discovery v0.2.4                                                                                                                                                                          
  [3da002f7] ColorTypes v0.11.5                                                                                                                                                                                     
  [5ae59095] Colors v0.12.10                                                                                                                                                                                        
  [34da2185] Compat v4.15.0                                                                                                                                                                                         
  [a8cc5b0e] Crayons v4.1.1                                                                                                                                                                                         
  [9a962f9c] DataAPI v1.16.0                                                                                                                                                                                        
  [a93c6f00] DataFrames v1.6.1                                                                                                                                                                                      
  [864edb3b] DataStructures v0.18.20                                                                                                                                                                                
  [e2d170a0] DataValueInterfaces v1.0.0                                                                                                                                                                             
  [e2ba6199] ExprTools v0.1.10                                                                                                                                                                                      
  [53c48c17] FixedPointNumbers v0.8.4                                                                                                                                                                               
  [0c68f7d7] GPUArrays v10.1.0                                                                                                                                                                                      
  [46192b85] GPUArraysCore v0.1.6                                                                                                                                                                                   
  [61eb1bfa] GPUCompiler v0.26.4                                                                                                                                                                                    
  [842dd82b] InlineStrings v1.4.0                                                                                                                                                                                   
  [41ab1584] InvertedIndices v1.3.0                                                                                                                                                                                 
  [82899510] IteratorInterfaceExtensions v1.0.0                                                                                                                                                                     
  [692b3bcd] JLLWrappers v1.5.0                                                                                                                                                                                     
  [63c18a36] KernelAbstractions v0.9.18                                                                                                                                                                             
⌅ [929cbde3] LLVM v6.6.3                                                                                                                                                                                            
  [8b046642] LLVMLoopInfo v1.0.0                                                                                                                                                                                    
  [b964fa9f] LaTeXStrings v1.3.1                                                                                                                                                                                    
  [1914dd2f] MacroTools v0.5.13                                                                                                                                                                                     
  [e1d29d7a] Missings v1.2.0                                                                                                                                                                                        
  [5da4648a] NVTX v0.3.4                                                                                                                                                                                            
  [bac558e1] OrderedCollections v1.6.3                                                                                                                                                                              
  [69de0a69] Parsers v2.8.1                                                                                                                                                                                         
  [2dfb63ee] PooledArrays v1.4.3                                                                                                                                                                                    
  [aea7be01] PrecompileTools v1.2.1                                                                                                                                                                                 
  [21216c6a] Preferences v1.4.3                                                                                                                                                                                     
  [08abe8d2] PrettyTables v2.3.1                                                                                                                                                                                    
  [74087812] Random123 v1.7.0                                                                                                                                                                                       
  [e6cf234a] RandomNumbers v1.5.3                                                                                                                                                                                   
  [189a3867] Reexport v1.2.2                                                                                                                                                                                        
  [ae029012] Requires v1.3.0                                                                                                                                                                                        
  [6c6a2e73] Scratch v1.2.1                                                                                                                                                                                         
  [91c51154] SentinelArrays v1.4.1                                                                                                                                                                                  
  [a2af1166] SortingAlgorithms v1.2.1                                                                                                                                                                               
  [90137ffa] StaticArrays v1.9.3                                                                                                                                                                                    
  [1e83bf80] StaticArraysCore v1.4.2
  [892a3eda] StringManipulation v0.3.4
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.11.1
  [a759f4b9] TimerOutputs v0.5.23
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [4ee394cb] CUDA_Driver_jll v0.8.1+0
⌅ [76a88914] CUDA_Runtime_jll v0.12.1+0
  [9c1d0b0a] JuliaNVTXCallbacks_jll v0.2.1+0
  [dad2f222] LLVMExtra_jll v0.0.29+0
  [e98f9f5b] NVTX_jll v3.1.0+2
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2

And the project environment is

Project Raven v0.1.0-pre                                                                                                                                                                                            
Status `/tmp/lucas/Raven.jl/Manifest.toml`                                                                                                                                                                          
  [1520ce14] AbstractTrees v0.4.5                                                                                                                                                                                   
  [79e6a3ab] Adapt v4.0.4                                                                                                                                                                                           
  [a9b6321e] Atomix v0.1.0                                                                                                                                                                                          
  [fa961155] CEnum v0.5.0                                                                                                                                                                                           
  [34da2185] Compat v4.15.0                                                                                                                                                                                         
  [ffbed154] DocStringExtensions v0.9.3                                                                                                                                                                             
  [46192b85] GPUArraysCore v0.1.6                                                                                                                                                                                   
  [692b3bcd] JLLWrappers v1.5.0                                                                                                                                                                                     
  [63c18a36] KernelAbstractions v0.9.18                                                                                                                                                                             
⌅ [929cbde3] LLVM v6.6.3                                                                                                                                                                                            
  [da04e1cc] MPI v0.20.19                                                                                                                                                                                           
  [3da0fdf6] MPIPreferences v0.1.11                                                                                                                                                                                 
  [1914dd2f] MacroTools v0.5.13                                                                                                                                                                                     
  [c5182250] OneDimensionalNodes v1.1.0                                                                                                                                                                             
  [7d669430] P4est v0.4.13                                                                                                                                                                                          
  [f636fe8e] P4estTypes v0.1.3                                                                                                                                                                                      
  [eebad327] PkgVersion v0.3.3                                                                                                                                                                                      
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [3cdcf5f2] RecipesBase v1.3.4
  [189a3867] Reexport v1.2.2
  [ae029012] Requires v1.3.0
  [90137ffa] StaticArrays v1.9.3
  [1e83bf80] StaticArraysCore v1.4.2
  [013be700] UnsafeAtomics v0.2.1
  [d80eeb9a] UnsafeAtomicsLLVM v0.1.3
  [e33a78d0] Hwloc_jll v2.10.0+0
  [dad2f222] LLVMExtra_jll v0.0.29+0
  [7cb0a576] MPICH_jll v4.2.1+1
  [f1f71cc9] MPItrampoline_jll v5.3.3+1
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
⌅ [fe0851c0] OpenMPI_jll v4.1.6+0
  [6b5a15aa] P4est_jll v2.8.1+2
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
vchuravy commented 1 month ago

Ah shoot. I had expected it to be related to https://github.com/JuliaGPU/CUDA.jl/pull/2336

But this looks much more like a core Julia bug.

vchuravy commented 1 month ago

I am going to try and creduce this.

vchuravy commented 1 month ago

So it does belong here since on a machine without CUDA it is not reproducible.