search

EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator
https://enzyme.mit.edu
MIT License
458 stars 66 forks source link

Reverse gradient works with CPU + KA but errors out after ~minutes with error on CUDA + KA #2133

Open roflmaostc opened 8 hours ago

roflmaostc commented 8 hours ago

Hi,

the following is veeeery slow on CUDA and also errors out:

julia> using CUDA, Enzyme, RadonKA, DifferentiationInterface

julia> function main()
                  arr = Array(rand(Float32, 256, 256, 32))
                  angles = Array(Float32.(range(0, 2π, 200)))

                  f(x, angles) = sum(radon(x, angles))
                  radon(arr, angles)

                  dx = 0 .* arr
                  x = copy(arr)
                  autodiff(Reverse, f, Active, Duplicated(x, dx), Const(angles))

                  return 0
              end
main (generic function with 1 method)

julia> main()
0

julia> function main()
                  arr = CuArray(rand(Float32, 256, 256, 32))
                  angles = CuArray(Float32.(range(0, 2π, 200)))

                  f(x, angles) = sum(radon(x, angles))
                  radon(arr, angles)

                  dx = 0 .* arr
                  x = copy(arr)
                  autodiff(Reverse, f, Active, Duplicated(x, dx), Const(angles))

                  return 0
              end

julia> main()
ERROR: Constant memory is stored (or returned) to a differentiable variable.
As a result, Enzyme cannot provably ensure correctness and throws this error.
This might be due to the use of a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Runtime-Activity).
If Enzyme should be able to prove this use non-differentable, open an issue!
To work around this issue, either:
 a) rewrite this variable to not be conditionally active (fastest, but requires a code change), or
 b) set the Enzyme mode to turn on runtime activity (e.g. autodiff(set_runtime_activity(Reverse), ...) ). This will maintain correctness, but may slightly reduce performance.
Mismatched activity for:   store {} addrspace(10)* %1, {} addrspace(10)* addrspace(10)* %.fca.2.gep, align 8, !dbg !1368, !noalias !1335 const val: {} addrspace(10)* %1
 value=Unknown object of type CuArray{Float32, 1, CUDA.DeviceMemory}
 llvalue={} addrspace(10)* %1

Stacktrace:
 [1] #context!#990
   @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/state.jl:168
 [2] context!
   @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/state.jl:163
 [3] unsafe_copyto!
   @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:575

Stacktrace:
  [1] #context!#990
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/state.jl:168 [inlined]
  [2] context!
    @ ~/.julia/packages/CUDA/2kjXI/lib/cudadrv/state.jl:163 [inlined]
  [3] unsafe_copyto!
    @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:575
  [4] copyto!
    @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:517 [inlined]
  [5] copyto!
    @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:521 [inlined]
  [6] copy
    @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:179 [inlined]
  [7] CuArray
    @ ~/.julia/packages/CUDA/2kjXI/src/array.jl:422 [inlined]
  [8] _radon
    @ ~/.julia/packages/RadonKA/FXHjL/src/radon.jl:145
  [9] #radon#10
    @ ~/.julia/packages/RadonKA/FXHjL/src/radon.jl:126 [inlined]
 [10] radon
    @ ~/.julia/packages/RadonKA/FXHjL/src/radon.jl:123
 [11] f
    @ ./REPL[27]:5 [inlined]
 [12] diffejulia_f_184005wrap
    @ ./REPL[27]:0
 [13] macro expansion
    @ ~/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8398 [inlined]
 [14] enzyme_call
    @ ~/.julia/packages/Enzyme/RTS5U/src/compiler.jl:7950 [inlined]
 [15] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/RTS5U/src/compiler.jl:7723 [inlined]
 [16] autodiff
    @ ~/.julia/packages/Enzyme/RTS5U/src/Enzyme.jl:491 [inlined]
 [17] autodiff
    @ ~/.julia/packages/Enzyme/RTS5U/src/Enzyme.jl:512 [inlined]
 [18] main()
    @ Main ./REPL[27]:11
 [19] top-level scope
    @ REPL[28]:1

julia> versioninfo()
Julia Version 1.10.6
Commit 67dffc4a8ae (2024-10-28 12:23 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 24 × AMD Ryzen 9 5900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 24 default, 0 interactive, 12 GC (on 24 virtual cores)
Environment:
  JULIA_NUM_THREADS = 24
  JULIA_MAX_NUM_PRECOMPILE_FILES = 100

julia> CUDA.versioninfo()
CUDA runtime 12.6, artifact installation
CUDA driver 12.4
NVIDIA driver 550.120.0

CUDA libraries: 
- CUBLAS: 12.6.4
- CURAND: 10.3.7
- CUFFT: 11.3.0
- CUSOLVER: 11.7.1
- CUSPARSE: 12.5.4
- CUPTI: 2024.3.2 (API 24.0.0)
- NVML: 12.0.0+550.120

Julia packages: 
- CUDA: 5.5.2
- CUDA_Driver_jll: 0.10.4+0
- CUDA_Runtime_jll: 0.15.5+0

Toolchain:
- Julia: 1.10.6
- LLVM: 15.0.7

1 device:
  0: NVIDIA GeForce RTX 3060 (sm_86, 11.012 GiB / 12.000 GiB available)
wsmoses commented 6 hours ago

looks like the constructor rule for CuArray here https://github.com/roflmaostc/RadonKA.jl/blob/5960b7cd3d4d845a92e85ea1112cdf89f631b81c/src/radon.jl#L145C4-L145C40 isn't getting triggered for some reason. File an issue on CUDA.jl? (ideally with a self contained function we can add a test to make sure it doesn't pop up in the future)

roflmaostc commented 2 hours ago

Now it got even worse

using CUDA, Enzyme

function f(x, y)
    y2 = similar(x, (size(y, 1),))
    y2 .= y 
    return sum(x .* y2)
end

x = CUDA.rand(10)
y = 1:10
dx = make_zero(x)
@show f(x,y)
autodiff(Reverse, f, Active, Duplicated(x, dx), Const(y))

julia> include("/tmp/lel2.jl")
f(x, y) = 23.677639f0
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:4412: DIFFE_TYPE GradientUtils::getDiffeType(llvm::Value*, bool) const: Assertion `unnecessaryValuesP' failed.

[583243] signal (6.-6): Aborted
in expression starting at /tmp/lel2.jl:15
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x74dbba22881a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
getDiffeType at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:4412
EnzymeGradientUtilsGetDiffeType at /home/felix/.julia/packages/Enzyme/RTS5U/src/api.jl:742
unknown function (ip: 0x74db5f12bba0)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme_custom_setup_args at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:117
unknown function (ip: 0x74db5f11c0ca)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
aug_fwd_mi at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:781 [inlined]
aug_fwd_mi at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:774 [inlined]
has_aug_fwd_rule at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:878 [inlined]
enzyme_custom_diffuse at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:1602
unknown function (ip: 0x74db5f117b7f)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme_custom_diffuse_cfunc at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/llvmrules.jl:105
unknown function (ip: 0x74db68d374b7)
operator() at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:383 [inlined]
__invoke_impl<bool, EnzymeRegisterDiffUseCallHandler(char*, CustomFunctionDiffUse)::<lambda(const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&)>&, const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/invoke.h:60 [inlined]
__invoke_r<bool, EnzymeRegisterDiffUseCallHandler(char*, CustomFunctionDiffUse)::<lambda(const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&)>&, const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/invoke.h:113 [inlined]
_M_invoke at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/std_function.h:291
operator() at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/std_function.h:622 [inlined]
is_use_directly_needed_in_reverse at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.cpp:464
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:419
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
computeForwardingProperties at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8733
computeGuaranteedFrees at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:9276
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2344
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5303
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6479
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4305
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:631
EnzymeCreatePrimalAndGradient at /home/felix/.julia/packages/Enzyme/RTS5U/src/api.jl:253
unknown function (ip: 0x74db5f115fae)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme! at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:4186
unknown function (ip: 0x74db5f114398)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#codegen#19015 at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:7338
codegen at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:6146 [inlined]
_thunk at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8468
_thunk at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8468 [inlined]
cached_compilation at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8509 [inlined]
thunkbase at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8641
unknown function (ip: 0x74db5ff30600)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#s2103#19072 at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8778 [inlined]
#s2103#19072 at ./none:0
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
GeneratedFunctionStub at ./boot.jl:602
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_call_staged at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2889
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45362.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:504
jl_compile_method_internal at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2481 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2887 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2076
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
_include at ./loading.jl:2136
include at ./client.jl:494
unknown function (ip: 0x74dbb9a8af05)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
eval_user_input at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:150
repl_backend_loop at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:246
#start_repl_backend#46 at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:231
start_repl_backend at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:228
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#run_repl#59 at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:389
run_repl at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:375
jfptr_run_repl_91949.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#1013 at ./client.jl:437
jfptr_YY.1013_82918.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_latest at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
run_main_repl at ./client.jl:421
exec_options at ./client.jl:338
_start at ./client.jl:557
jfptr__start_82944.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
unknown function (ip: 0x74dbba22a1c9)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 196795431 (Pool: 196528539; Big: 266892); GC: 135
[1]    583243 IOT instruction (core dumped)  julia

``