EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator
https://enzyme.mit.edu
MIT License
455 stars 64 forks source link

Assertion failed error #1011

Closed swilliamson7 closed 1 year ago

swilliamson7 commented 1 year ago

There's an ocean model I'd like to be able to use with Enzyme, but I've been hitting an "assertion failed" error early on in the integration function. I've been working on creating a MWE but it's a bit tricky as there are a lot of moving parts. I'll continue working on it, but in the meantime I thought I'd post the error in the hopes you might know more about what's going on. I saw someone else posted an issue that looked similar to what I've been seeing, at least to someone who doesn't develop Enzyme. I'm attaching the output as a text file as it's quite long

error_advection_coriolis_only5.txt

wsmoses commented 1 year ago

yup that's the same issue as https://github.com/EnzymeAD/Enzyme.jl/issues/1009

It was accidentally added by me as a consequence of making other errors nicer for julia (specifically turning them into runtime rather than compile time errors).

So it is in some sense now fixed by the linked PR, but unfortunately will require a JLL. bump to land here with the update before you can use it from Julia.

swilliamson7 commented 1 year ago

Amazing! I'll hold off on working on it until that update happens then. Is my understanding correct that I don't need to make a MWE in this case?

wsmoses commented 1 year ago

Correct! If you want to try it early you could use that custom jll process, but it's probably just easier to wait for the next bump.

cc @vchuravy

wsmoses commented 1 year ago

bumped jll, please retry!

swilliamson7 commented 1 year ago

Just retried my code and got a different error:

JIT session error: In graph -jitted-objectbuffer, section __TEXT,__text: relocation target "l___unnamed_142" at address 0x7972c7153 is out of range of Page21 fixup at 0x6ced90830 (_diffejulia_time_integration_debug_3447_inner_1wrap, 0x6ced8c308 + 0x4528)
JIT session error: In graph -jitted-objectbuffer, section __TEXT,__text: relocation target "l___unnamed_142" at address 0x9711bb153 is out of range of Page21 fixup at 0x8a8c84830 (_diffejulia_time_integration_debug_5764_inner_1wrap, 0x8a8c80308 + 0x4528)
JIT session error: In graph -jitted-objectbuffer, section __TEXT,__text: relocation target "l___unnamed_142" at address 0xb2e177153 is out of range of Page21 fixup at 0xa65c40830 (_diffejulia_time_integration_debug_5790_inner_1wrap, 0xa65c3c308 + 0x4528)
ERROR: LoadError: LLVM error: Failed to materialize symbols: { (main, { _diffejulia_time_integration_debug_5790_inner_1wrap }) }
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/LLVM/Od0DH/src/executionengine/utils.jl:32 [inlined]
  [2] lookup
    @ ~/.julia/packages/LLVM/Od0DH/src/orcv2.jl:228 [inlined]
  [3] lookup
    @ ~/.julia/packages/Enzyme/Rut8C/src/compiler/orcv2.jl:262 [inlined]
  [4] _link(job::GPUCompiler.CompilerJob{Enzyme.Compiler.EnzymeTarget, Enzyme.Compiler.EnzymeCompilerParams}, ::Tuple{LLVM.Module, String, Nothing, DataType})
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Rut8C/src/compiler.jl:9670
  [5] cached_compilation
    @ ~/.julia/packages/Enzyme/Rut8C/src/compiler.jl:9726 [inlined]
  [6] (::Enzyme.Compiler.var"#475#476"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{Bool, Bool}, Int64, Bool, Bool, UInt64, DataType})(ctx::LLVM.Context)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/Rut8C/src/compiler.jl:9788
  [7] JuliaContext(f::Enzyme.Compiler.var"#475#476"{DataType, DataType, DataType, Enzyme.API.CDerivativeMode, Tuple{Bool, Bool}, Int64, Bool, Bool, UInt64, DataType})
    @ GPUCompiler ~/.julia/packages/GPUCompiler/kwkKA/src/driver.jl:58
  [8] #s260#474
    @ ~/.julia/packages/Enzyme/Rut8C/src/compiler.jl:9743 [inlined]
  [9] var"#s260#474"(FA::Any, A::Any, TT::Any, Mode::Any, ModifiedBetween::Any, width::Any, ReturnPrimal::Any, ShadowInit::Any, World::Any, ABI::Any, ::Any, #unused#::Type, #unused#::Type, #unused#::Type, tt::Any, #unused#::Type, #unused#::Type, #unused#::Type, #unused#::Type, #unused#::Type, #unused#::Any)
    @ Enzyme.Compiler ./none:0
 [10] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any})
    @ Core ./boot.jl:582
 [11] autodiff
    @ ~/.julia/packages/Enzyme/Rut8C/src/Enzyme.jl:207 [inlined]
 [12] autodiff
    @ ~/.julia/packages/Enzyme/Rut8C/src/Enzyme.jl:236 [inlined]
 [13] autodiff
    @ ~/.julia/packages/Enzyme/Rut8C/src/Enzyme.jl:222 [inlined]
 [14] run_enzyme(#unused#::Type{Float32}, P::Parameter)
    @ Main ~/Documents/GitHub/ShallowWaters.jl/src/advection_coriolis_only.jl:1905
 [15] #run_enzyme#48
    @ ~/Documents/GitHub/ShallowWaters.jl/src/advection_coriolis_only.jl:1881 [inlined]
 [16] top-level scope
    @ ~/Documents/GitHub/ShallowWaters.jl/src/advection_coriolis_only.jl:1911
in expression starting at /Users/swilliamson/Documents/GitHub/ShallowWaters.jl/src/advection_coriolis_only.jl:1911

I've seen this before, messing around with the code that led to it is how I found the assertion error. Could you offer insight into what this one might be saying? I can't really tell what the error message actually is

wsmoses commented 1 year ago

weird, open a new issue? cc @vchuravy