search

EnzymeAD / Enzyme.jl

Julia bindings for the Enzyme automatic differentiator
https://enzyme.mit.edu
MIT License
439 stars 62 forks source link

Enzyme broken for Windows on Julia v1.10 #1236

Closed jeremiedb closed 6 months ago

jeremiedb commented 8 months ago

The following minimal script hangs forever when launched on Windows machine with Julia 1.10. It works fine with Julia 1.9.4 and 1.8.3: 

using Enzyme
@info "starting"
f1(x) = x*x
autodiff(Reverse, f1, Active(1.0))
@info "done"
PS C:\Users\jerem\OneDrive\github\ADTests.jl> julia --project=@. --threads=1 .\experiments\debug.jl
[ Info: starting

I noticed some work related to Julia v1.10 (https://github.com/EnzymeAD/Enzyme.jl/pull/1231) and that Windows CI on Julia 1.10 is failing, so perhaps this was already on the radar?

leerosenthalj commented 8 months ago

I am encountering the same issue, also with Julia 1.10 on Windows, with a similar MWE.

vchuravy commented 8 months ago

If you can get a backtrace from where the hang is coming from there might be something we can intuit from it.

ArnoStrouwen commented 7 months ago

Interrupting Enzyme with Ctrl+C is not working. I created a dump file of the hanging Julia process on Windows, which might contain something useful:


************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Users\arno\AppData\Local\Temp\julia (2).DMP]
User Mini Dump File with Full Memory: Only application data is available

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 
Windows 10 Version 19045 MP (24 procs) Free x64
Product: WinNt, suite: SingleUserTS
Edition build lab: 19041.1.amd64fre.vb_release.191206-1406
Debug session time: Sat Jan 27 19:42:09.000 2024 (UTC + 1:00)
System Uptime: 0 days 1:14:15.392
Process Uptime: 0 days 0:01:26.000
................................................................
..................

+------------------------------------------------------------------------+
| This target supports Hardware-enforced Stack Protection. A HW based    |
| "Shadow Stack" may be available to assist in debugging and analysis.   |
| See aka.ms/userhsp for more info.                                      |
|                                                                        |
| dps @ssp                                                               |
|                                                                        |
+------------------------------------------------------------------------+

For analysis of this file, run !analyze -v
ntdll!NtWaitForSingleObject+0x14:
00007fff`a6e8d064 c3              ret
0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1671

    Key  : Analysis.Elapsed.mSec
    Value: 1665

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 0

    Key  : Analysis.Init.CPU.mSec
    Value: 342

    Key  : Analysis.Init.Elapsed.mSec
    Value: 110180

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 118

    Key  : Failure.Bucket
    Value: BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

    Key  : Failure.Hash
    Value: {656a880b-7201-663a-ee71-fadf484d9a93}

    Key  : Timeline.OS.Boot.DeltaSec
    Value: 4455

    Key  : Timeline.Process.Start.DeltaSec
    Value: 86

    Key  : WER.OS.Branch
    Value: vb_release

    Key  : WER.OS.Version
    Value: 10.0.19041.1

    Key  : WER.Process.Version
    Value: 1.10.0.0

FILE_IN_CAB:  julia (2).DMP

NTGLOBALFLAG:  0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS:  0

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  00000920

PROCESS_NAME:  julia.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE_STR:  80000003

STACK_TEXT:  
00000099`0b9faca8 00007fff`a47330ce     : 00000289`ce550000 00007fff`a6e147b1 00000289`fd438700 00007fff`a6e15ba1 : ntdll!NtWaitForSingleObject+0x14
00000099`0b9facb0 00007fff`89632e49     : 00000289`d4fb6290 00000000`00000001 00000000`00000000 00000000`00000340 : KERNELBASE!WaitForSingleObjectEx+0x8e
00000099`0b9fad50 00007fff`356a3244     : 00000289`d08eacf0 00000000`00000000 00000000`00000000 00007fff`89633129 : libwinpthread_1!pthread_mutex_lock+0x99
00000099`0b9fada0 00000000`6f51b593     : 00000289`fd4386f0 00000289`fcf0da60 00000289`fd438708 00000099`0b9faf00 : libjulia_codegen!LLVMExtraLPMAddLowerSIMDLoopPass_impl+0x28fb4
00000099`0b9fae00 00000000`6f52cd85     : 00000289`fd438708 00000289`c0103e48 00007fff`356bf5c1 00000000`00000018 : libLLVM_15jl!ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x243
00000099`0b9faea0 00007fff`355e28a7     : 00000289`bf8fcaa0 00000289`d08e75c0 00000289`bf8fc560 00007fff`a6e147b1 : libLLVM_15jl!ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x185
00000099`0b9faf60 00000000`6f57200c     : 00000289`d08e7878 00000289`00000000 00000289`d08ea690 00000289`c055d370 : libjulia_codegen!jl_jit_total_bytes_impl+0xe47
00000099`0b9fb030 00000000`6f53a48c     : 00000000`00000018 00000289`d08e7610 00000000`715b6d00 00007fff`a5c99d40 : libLLVM_15jl!ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE+0xac
00000099`0b9fb110 00000000`6f784fbd     : 00000099`0b9fb1b8 00007fff`3bc5e21c 00000000`00000018 00000289`d08e7610 : libLLVM_15jl!ZN4llvm3orc19MaterializationTask3runEv+0x2c
00000099`0b9fb150 00000000`6f52ab3c     : 00000289`d08f32b0 00000289`bf8fcaa0 00000289`bfa52d88 00000000`00000000 : libLLVM_15jl!ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_+0x1d
00000099`0b9fb190 00000000`6f52842e     : 00000289`ce552ac0 00000289`caa69410 00000289`d09757a0 00007fff`a6e1b86b : libLLVM_15jl!ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv+0x9c
00000099`0b9fb200 00000000`6f55bc8d     : 00000289`d08e7580 00000099`0b9fb3a8 00000099`0b9fb3b0 00000099`0b9fb3c0 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE+0xb2e
00000099`0b9fb380 00000000`6f529999     : 00000000`00000000 00007fff`a6e10000 00000002`ca0959e8 00007fff`3bc5e21c : libLLVM_15jl!ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE+0xdd
00000099`0b9fb400 00000000`6f52bd44     : 00000000`015c0657 00000000`000000e2 00000289`fcf238f0 00000000`6f5899dc : libLLVM_15jl!ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE+0xb19
00000099`0b9fb550 00000000`6f52c297     : 00000289`d08e7878 00000000`00000000 00000289`d4fa9880 00000289`d4fa9c30 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE+0x3a4
00000099`0b9fb660 00007fff`355e4cd6     : 00000289`ce540000 00000289`d08e7580 00000099`0b9fb8e0 00000000`00000011 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE+0x1d7
00000099`0b9fb780 00007fff`355eb56a     : 00000289`d08e9708 00000000`00000000 00000000`00000000 00000000`00000030 : libjulia_codegen!jl_jit_total_bytes_impl+0x3276
00000099`0b9fb9f0 00007fff`355ec879     : 00000289`c919a1f8 00000099`0b9fbcf0 00000099`0b9fbd30 00000099`0b9fbd90 : libjulia_codegen!jl_compile_extern_c_impl+0x234a
00000099`0b9fbbb0 00007fff`355ed7eb     : 00000289`d3ff65b0 00000289`bf4d8d00 ffffffff`ffffffff 00000289`c292d1e0 : libjulia_codegen!jl_compile_extern_c_impl+0x3659
00000099`0b9fc3d0 00007fff`3cfb797f     : 00007fff`2d3d66f0 00000289`d07769b8 00000000`0000026b 00007fff`a4760fbb : libjulia_codegen!jl_generate_fptr_impl+0x2ab
00000099`0b9fc4c0 00007fff`3cfbb15f     : 00007fff`7ac64200 00007fff`3cfbb10e 00007fff`00000102 00000000`00000002 : libjulia_internal!jl_normalize_to_compilable_mi+0x1cf
00000099`0b9fc570 00007fff`3cfd626b     : 00000099`0b9fc650 00007fff`3cfd626b 00000000`00000001 00000000`00000030 : libjulia_internal!ijl_apply_generic+0x14f
00000099`0b9fc600 00007fff`3cfd5d9d     : 00000289`fdca8608 00007fff`3d00ce84 00000000`00000002 00000289`d0b00080 : libjulia_internal!ijl_init_restored_module+0x9ab
00000099`0b9fc690 00007fff`3cfd6929     : 00000289`c39f2770 00000099`0b9fc950 00000000`00000302 00000000`00000003 : libjulia_internal!ijl_init_restored_module+0x4dd
00000099`0b9fc730 00007fff`3cfd772f     : 00000289`c39f2770 00000099`0b9fc950 00000289`fd726e86 00000000`00000001 : libjulia_internal!ijl_init_restored_module+0x1069
00000099`0b9fc930 00007fff`3cff306c     : 00000289`c1de4190 00007fff`3d00ce84 00000000`00000038 00000001`00000000 : libjulia_internal!jl_interpret_toplevel_thunk+0xef
00000099`0b9fca10 00007fff`3cff39df     : 00000000`00000005 00000099`0b9fcbe0 00000099`0b9fcb48 00000001`00000010 : libjulia_internal!jl_toplevel_eval_flex+0x35c
00000099`0b9fcb50 00007fff`3cff4a00     : 00000289`c1d97eb0 00000289`c40277f0 00000289`c40277f0 00000289`c1de40b0 : libjulia_internal!jl_toplevel_eval_flex+0xccf
00000099`0b9fcc90 00007fff`287a6e44     : 00007fff`2d3d66f0 00000289`c1de40b0 00007fff`276c7835 00000000`00000000 : libjulia_internal!ijl_toplevel_eval_in+0x90
00000099`0b9fce20 00007fff`292d8ce0     : 00007fff`293bd270 00007fff`2c28e1a0 00000000`00000eb8 00000000`00000002 : sys+0x1286e44
00000099`0b9fd180 00007fff`28054ec4     : 00000099`0b9fd400 00000289`d0b00080 00000289`d0b00080 00000099`0b9fd340 : sys+0x1db8ce0
00000099`0b9fd2c0 00007fff`287a491d     : 00007fff`29314650 00000289`d0b00080 00000289`d45843a0 00000000`00007af8 : sys+0xb34ec4
00000099`0b9fd320 00007fff`28053c8d     : 00007fff`2d86ffd0 00007fff`3d00ce84 00000000`00000001 00007fff`2d8705c0 : sys+0x128491d
00000099`0b9fd3e0 00007fff`27c9c0e3     : 00000000`00000000 00000000`00000000 00000000`00007af8 00000000`00000000 : sys+0xb33c8d
00000099`0b9fd540 00007fff`28f3dcc5     : 00000099`0b9fd6c8 00007fff`2c754dd0 00007fff`319c9850 00007fff`2c8ed8c0 : sys+0x77c0e3
00000099`0b9fd660 00007fff`28f17a08     : 00000099`0b9fd8e0 00007fff`3d280440 00007fff`2a21f670 00000289`d49040a0 : sys+0x1a1dcc5
00000099`0b9fd6a0 00007fff`278b4e8b     : 00000000`00000000 00000289`d0b00080 00000099`0b9fd8d8 00000000`00000002 : sys+0x19f7a08
00000099`0b9fd800 00007fff`3cfc695b     : 00000000`00007af8 00000289`00000006 00000000`00000000 00000000`00000001 : sys+0x394e8b
00000099`0b9fd850 00007fff`292d3f13     : 00000000`00000081 00000000`00000000 00000000`00000080 00000000`00000000 : libjulia_internal!jl_f__call_latest+0x3b
00000099`0b9fd8a0 00007fff`292c5e87     : 00000289`0000000f 00000000`00000000 00000000`00000004 00000000`00000000 : sys+0x1db3f13
00000099`0b9fdc00 00007fff`28f3d6b4     : 00000000`00000000 00000099`0b9ff060 00000099`0b9fefb8 00000099`0b9ff050 : sys+0x1da5e87
00000099`0b9ff060 00007fff`283f9f7d     : 00000289`ce364d18 00000289`d0b00010 00000000`0000004a 00007fff`a4e65ec0 : sys+0x1a1d6b4
00000099`0b9ff370 00007fff`3d02759a     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : sys+0xed9f7d
00000099`0b9ff3b0 00007fff`3d028059     : 00007fff`965e4700 00000099`0b9ff6f8 00000000`00000010 00007fff`965d1d74 : libjulia_internal!ijl_call2+0x36a
00000099`0b9ff6d0 00007ff7`91d11ab2     : 00000289`00000000 00000289`00000000 00000289`ce3541d0 00000099`0b9f0000 : libjulia_internal!jl_repl_entrypoint+0x79
00000099`0b9ff730 00007fff`a4e67344     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : julia+0x1ab2
00000099`0b9ff7f0 00007fff`a6e426b1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000099`0b9ff820 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21

SYMBOL_NAME:  libwinpthread_1+99

MODULE_NAME: libwinpthread_1

IMAGE_NAME:  libwinpthread-1.dll

STACK_COMMAND:  dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb

FAILURE_BUCKET_ID:  BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

IMAGE_VERSION:  1.0.0.0

FAILURE_ID_HASH:  {656a880b-7201-663a-ee71-fadf484d9a93}

Followup:     MachineOwner
---------
wsmoses commented 7 months ago

That looks like it is stuck in the LLVM pass manager itself (and with a lock).

We may be able to use the Julia internal wrapper for the llvm pass pipeline. However, it needs this to be backported and then integrated in: https://github.com/JuliaLang/julia/pull/52544

Would JuliaLang be able backport that? Also, if that were sufficient to fix, obviously that would require waiting til the next Julia 1.10 release.

On Sat, Jan 27, 2024 at 1:56 PM Arno Strouwen @.***> wrote:

Interrupting Enzyme with Ctrl+C is not working. I created a dump file of the hanging Julia process on Windows, which might contain something useful:

***** Preparing the environment for Debugger Extensions Gallery repositories ** ExtensionRepository : Implicit UseExperimentalFeatureForNugetShare : true AllowNugetExeUpdate : true AllowNugetMSCredentialProviderInstall : true AllowParallelInitializationOfLocalRepositories : true

-- Configuring repositories ----> Repository : LocalInstalled, Enabled: true ----> Repository : UserExtensions, Enabled: true

Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

***** Waiting for Debugger Extensions Gallery to Initialize **

Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds ----> Repository : UserExtensions, Enabled: true, Packages count: 0 ----> Repository : LocalInstalled, Enabled: true, Packages count: 36

Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64 Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Users\arno\AppData\Local\Temp\julia (2).DMP] User Mini Dump File with Full Memory: Only application data is available

***** Path validation summary ** Response Time (ms) Location Deferred srv Symbol search path is: srv Executable search path is: Windows 10 Version 19045 MP (24 procs) Free x64 Product: WinNt, suite: SingleUserTS Edition build lab: 19041.1.amd64fre.vb_release.191206-1406 Debug session time: Sat Jan 27 19:42:09.000 2024 (UTC + 1:00) System Uptime: 0 days 1:14:15.392 Process Uptime: 0 days 0:01:26.000 ................................................................ ..................

+------------------------------------------------------------------------+ This target supports Hardware-enforced Stack Protection. A HW based "Shadow Stack" may be available to assist in debugging and analysis. See aka.ms/userhsp for more info.
dps @ssp

+------------------------------------------------------------------------+

For analysis of this file, run !analyze -v ntdll!NtWaitForSingleObject+0x14: 00007fff`a6e8d064 c3 ret 0:000> !analyze -v

  • *
  • Exception Analysis *
  • *

KEY_VALUES_STRING: 1

Key  : Analysis.CPU.mSec
Value: 1671

Key  : Analysis.Elapsed.mSec
Value: 1665

Key  : Analysis.IO.Other.Mb
Value: 0

Key  : Analysis.IO.Read.Mb
Value: 0

Key  : Analysis.IO.Write.Mb
Value: 0

Key  : Analysis.Init.CPU.mSec
Value: 342

Key  : Analysis.Init.Elapsed.mSec
Value: 110180

Key  : Analysis.Memory.CommitPeak.Mb
Value: 118

Key  : Failure.Bucket
Value: BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

Key  : Failure.Hash
Value: {656a880b-7201-663a-ee71-fadf484d9a93}

Key  : Timeline.OS.Boot.DeltaSec
Value: 4455

Key  : Timeline.Process.Start.DeltaSec
Value: 86

Key  : WER.OS.Branch
Value: vb_release

Key  : WER.OS.Version
Value: 10.0.19041.1

Key  : WER.Process.Version
Value: 1.10.0.0

FILE_IN_CAB: julia (2).DMP

NTGLOBALFLAG: 0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS: 0

EXCEPTION_RECORD: (.exr -1) ExceptionAddress: 0000000000000000 ExceptionCode: 80000003 (Break instruction exception) ExceptionFlags: 00000000 NumberParameters: 0

FAULTING_THREAD: 00000920

PROCESS_NAME: julia.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

EXCEPTION_CODE_STR: 80000003

STACK_TEXT: 000000990b9faca8 00007fffa47330ce : 00000289ce550000 00007fffa6e147b1 00000289fd438700 00007fffa6e15ba1 : ntdll!NtWaitForSingleObject+0x14 000000990b9facb0 00007fff89632e49 : 00000289d4fb6290 0000000000000001 0000000000000000 0000000000000340 : KERNELBASE!WaitForSingleObjectEx+0x8e 000000990b9fad50 00007fff356a3244 : 00000289d08eacf0 0000000000000000 0000000000000000 00007fff89633129 : libwinpthread_1!pthread_mutex_lock+0x99 000000990b9fada0 000000006f51b593 : 00000289fd4386f0 00000289fcf0da60 00000289fd438708 000000990b9faf00 : libjulia_codegen!LLVMExtraLPMAddLowerSIMDLoopPass_impl+0x28fb4 000000990b9fae00 000000006f52cd85 : 00000289fd438708 00000289c0103e48 00007fff356bf5c1 0000000000000018 : libLLVM_15jl!ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x243 000000990b9faea0 00007fff355e28a7 : 00000289bf8fcaa0 00000289d08e75c0 00000289bf8fc560 00007fffa6e147b1 : libLLVM_15jl!ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE+0x185 000000990b9faf60 000000006f57200c : 00000289d08e7878 0000028900000000 00000289d08ea690 00000289c055d370 : libjulia_codegen!jl_jit_total_bytes_impl+0xe47 000000990b9fb030 000000006f53a48c : 0000000000000018 00000289d08e7610 00000000715b6d00 00007fffa5c99d40 : libLLVM_15jl!ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE+0xac 000000990b9fb110 000000006f784fbd : 000000990b9fb1b8 00007fff3bc5e21c 0000000000000018 00000289d08e7610 : libLLVM_15jl!ZN4llvm3orc19MaterializationTask3runEv+0x2c 000000990b9fb150 000000006f52ab3c : 00000289d08f32b0 00000289bf8fcaa0 00000289bfa52d88 0000000000000000 : libLLVM_15jl!ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7EEEvPvRS7+0x1d 000000990b9fb190 000000006f52842e : 00000289ce552ac0 00000289caa69410 00000289d09757a0 00007fffa6e1b86b : libLLVM_15jl!ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv+0x9c 000000990b9fb200 000000006f55bc8d : 00000289d08e7580 000000990b9fb3a8 000000990b9fb3b0 000000990b9fb3c0 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE+0xb2e 000000990b9fb380 000000006f529999 : 0000000000000000 00007fffa6e10000 00000002ca0959e8 00007fff3bc5e21c : libLLVM_15jl!ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE+0xdd 000000990b9fb400 000000006f52bd44 : 00000000015c0657 00000000000000e2 00000289fcf238f0 000000006f5899dc : libLLVM_15jl!ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE+0xb19 000000990b9fb550 000000006f52c297 : 00000289d08e7878 0000000000000000 00000289d4fa9880 00000289d4fa9c30 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE+0x3a4 000000990b9fb660 00007fff355e4cd6 : 00000289ce540000 00000289d08e7580 000000990b9fb8e0 0000000000000011 : libLLVM_15jl!ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE+0x1d7 000000990b9fb780 00007fff355eb56a : 00000289d08e9708 0000000000000000 0000000000000000 0000000000000030 : libjulia_codegen!jl_jit_total_bytes_impl+0x3276 000000990b9fb9f0 00007fff355ec879 : 00000289c919a1f8 000000990b9fbcf0 000000990b9fbd30 000000990b9fbd90 : libjulia_codegen!jl_compile_extern_c_impl+0x234a 000000990b9fbbb0 00007fff355ed7eb : 00000289d3ff65b0 00000289bf4d8d00 ffffffffffffffff 00000289c292d1e0 : libjulia_codegen!jl_compile_extern_c_impl+0x3659 000000990b9fc3d0 00007fff3cfb797f : 00007fff2d3d66f0 00000289d07769b8 000000000000026b 00007fffa4760fbb : libjulia_codegen!jl_generate_fptr_impl+0x2ab 000000990b9fc4c0 00007fff3cfbb15f : 00007fff7ac64200 00007fff3cfbb10e 00007fff00000102 0000000000000002 : libjulia_internal!jl_normalize_to_compilable_mi+0x1cf 000000990b9fc570 00007fff3cfd626b : 000000990b9fc650 00007fff3cfd626b 0000000000000001 0000000000000030 : libjulia_internal!ijl_apply_generic+0x14f 000000990b9fc600 00007fff3cfd5d9d : 00000289fdca8608 00007fff3d00ce84 0000000000000002 00000289d0b00080 : libjulia_internal!ijl_init_restored_module+0x9ab 000000990b9fc690 00007fff3cfd6929 : 00000289c39f2770 000000990b9fc950 0000000000000302 0000000000000003 : libjulia_internal!ijl_init_restored_module+0x4dd 000000990b9fc730 00007fff3cfd772f : 00000289c39f2770 000000990b9fc950 00000289fd726e86 0000000000000001 : libjulia_internal!ijl_init_restored_module+0x1069 000000990b9fc930 00007fff3cff306c : 00000289c1de4190 00007fff3d00ce84 0000000000000038 0000000100000000 : libjulia_internal!jl_interpret_toplevel_thunk+0xef 000000990b9fca10 00007fff3cff39df : 0000000000000005 000000990b9fcbe0 000000990b9fcb48 0000000100000010 : libjulia_internal!jl_toplevel_eval_flex+0x35c 000000990b9fcb50 00007fff3cff4a00 : 00000289c1d97eb0 00000289c40277f0 00000289c40277f0 00000289c1de40b0 : libjulia_internal!jl_toplevel_eval_flex+0xccf 000000990b9fcc90 00007fff287a6e44 : 00007fff2d3d66f0 00000289c1de40b0 00007fff276c7835 0000000000000000 : libjulia_internal!ijl_toplevel_eval_in+0x90 000000990b9fce20 00007fff292d8ce0 : 00007fff293bd270 00007fff2c28e1a0 0000000000000eb8 0000000000000002 : sys+0x1286e44 000000990b9fd180 00007fff28054ec4 : 000000990b9fd400 00000289d0b00080 00000289d0b00080 000000990b9fd340 : sys+0x1db8ce0 000000990b9fd2c0 00007fff287a491d : 00007fff29314650 00000289d0b00080 00000289d45843a0 0000000000007af8 : sys+0xb34ec4 000000990b9fd320 00007fff28053c8d : 00007fff2d86ffd0 00007fff3d00ce84 0000000000000001 00007fff2d8705c0 : sys+0x128491d 000000990b9fd3e0 00007fff27c9c0e3 : 0000000000000000 0000000000000000 0000000000007af8 0000000000000000 : sys+0xb33c8d 000000990b9fd540 00007fff28f3dcc5 : 000000990b9fd6c8 00007fff2c754dd0 00007fff319c9850 00007fff2c8ed8c0 : sys+0x77c0e3 000000990b9fd660 00007fff28f17a08 : 000000990b9fd8e0 00007fff3d280440 00007fff2a21f670 00000289d49040a0 : sys+0x1a1dcc5 000000990b9fd6a0 00007fff278b4e8b : 0000000000000000 00000289d0b00080 000000990b9fd8d8 0000000000000002 : sys+0x19f7a08 000000990b9fd800 00007fff3cfc695b : 0000000000007af8 0000028900000006 0000000000000000 0000000000000001 : sys+0x394e8b 000000990b9fd850 00007fff292d3f13 : 0000000000000081 0000000000000000 0000000000000080 0000000000000000 : libjulia_internal!jl_f__call_latest+0x3b 000000990b9fd8a0 00007fff292c5e87 : 000002890000000f 0000000000000000 0000000000000004 0000000000000000 : sys+0x1db3f13 000000990b9fdc00 00007fff28f3d6b4 : 0000000000000000 000000990b9ff060 000000990b9fefb8 000000990b9ff050 : sys+0x1da5e87 000000990b9ff060 00007fff283f9f7d : 00000289ce364d18 00000289d0b00010 000000000000004a 00007fffa4e65ec0 : sys+0x1a1d6b4 000000990b9ff370 00007fff3d02759a : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : sys+0xed9f7d 000000990b9ff3b0 00007fff3d028059 : 00007fff965e4700 000000990b9ff6f8 0000000000000010 00007fff965d1d74 : libjulia_internal!ijl_call2+0x36a 000000990b9ff6d0 00007ff791d11ab2 : 0000028900000000 0000028900000000 00000289ce3541d0 000000990b9f0000 : libjulia_internal!jl_repl_entrypoint+0x79 000000990b9ff730 00007fffa4e67344 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : julia+0x1ab2 000000990b9ff7f0 00007fffa6e426b1 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0x14 000000990b9ff820 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21

SYMBOL_NAME: libwinpthread_1+99

MODULE_NAME: libwinpthread_1

IMAGE_NAME: libwinpthread-1.dll

STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~0s; .ecxr ; kb

FAILURE_BUCKET_ID: BREAKPOINT_80000003_libwinpthread-1.dll!Unknown

OS_VERSION: 10.0.19041.1

BUILDLAB_STR: vb_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 1.0.0.0

FAILURE_ID_HASH: {656a880b-7201-663a-ee71-fadf484d9a93}

Followup: MachineOwner

— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme.jl/issues/1236#issuecomment-1913294486, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXEQTGX73MVBRJE5UPTYQVEXHAVCNFSM6AAAAABBK5SXLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTGI4TINBYGY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Larbino1 commented 7 months ago

Ahhh, I must be experiencing the same issue. Hangs on differentiating a simple function on my windows 10 PC, Ctrl+C doesn't work, recently updated to Julia 1.10, before which it was working.

salbalkus commented 7 months ago

I just started using Enzyme.jl with Julia and I'm experiencing the same issue as well.

RGonTheNoble commented 7 months ago

I tried using it on version 1.10.0 and 1.10.1, on Windows 11 and in both instances it just froze. I was running a simple script used in the example:

f1(x) = x*x
autodiff(Reverse, f1, Active(1.0))

Note that when running it on Kubuntu 22.04 it worked.

wsmoses commented 7 months ago

Yeah as far as we’ve investigated presently, Julia 1.10 on Windows causes one of our dependencies to lock up, so it doesn’t matter what the input is.

At present the best advice I can give is to downgrade to Julia 1.9 or switch to a different OS.

I hope this gets resolved in a later Julia patch, but it’s not something Enzyme devs have had cycles to debug the origin of, having enough core Enzyme work to more than fill our free time =/.

Help certainly wanted/appreciated.

On Sat, Feb 24, 2024 at 5:01 PM RGonTheNoble @.***> wrote:

I tried using it on version 1.10.0 and 1.10.1, on Windows 11 and in both instances it just froze. I was running a simple script used in the example: f1(x) = x*x autodiff(Reverse, f1, Active(1.0)) Note that when running it on Kubuntu 22.04 it worked.

— Reply to this email directly, view it on GitHub https://github.com/EnzymeAD/Enzyme.jl/issues/1236#issuecomment-1962774839, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJTUXEXMXCNKJKNWGYJ5XDYVKEP7AVCNFSM6AAAAABBK5SXLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRSG43TIOBTHE . You are receiving this because you commented.Message ID: <EnzymeAD/Enzyme. @.***>

wsmoses commented 6 months ago

Reduce to a pure bug in LLVM.jl.

Unfortunately nothing we can do here in Enzyme to fix, but feel free to bump LLVM.jl about a fix:

 using LLVM

function cpu_name()
    ccall(:jl_get_cpu_name, String, ())
end

function cpu_features()
    return ccall(:jl_get_cpu_features, String, ())
end

const modstr = """
; ModuleID = 'start'
source_filename = "start"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-w64-mingw32"

; Function Attrs: inaccessiblemem_or_argmemonly nofree
declare void @julia.safepoint(i64*) local_unnamed_addr #0

; Function Attrs: alwaysinline nofree
define [1 x double] @fwddiffejulia_identity_1243wrap(double %0, double %1) #1 !dbg !4 {
entry:
  %2 = call {}*** inttoptr (i64 140708411134816 to {}*** ()*)() #5
  %ptls_field3.i = getelementptr inbounds {}**, {}*** %2, i64 2
  %3 = bitcast {}*** %ptls_field3.i to i64***
  %ptls_load45.i = load i64**, i64*** %3, align 8, !tbaa !8, !alias.scope !12, !noalias !15
  %4 = getelementptr inbounds i64*, i64** %ptls_load45.i, i64 2
  %safepoint.i = load i64*, i64** %4, align 8, !tbaa !17, !alias.scope !19, !noalias !22
  fence syncscope("singlethread") seq_cst
  %5 = load volatile i64, i64* %safepoint.i, align 8, !dbg !24
  fence syncscope("singlethread") seq_cst
  %6 = insertvalue [1 x double] zeroinitializer, double %1, 0
  ret [1 x double] %6
}

; Function Attrs: inaccessiblemem_or_argmemonly
declare void @ijl_gc_queue_root({} addrspace(10)*) #2

; Function Attrs: allocsize(2)
declare noalias nonnull {} addrspace(10)* @ijl_gc_pool_alloc(i8*, i32, i32) #3

; Function Attrs: allocsize(1)
declare noalias nonnull {} addrspace(10)* @ijl_gc_big_alloc(i8*, i64) #4

; Function Attrs: allocsize(1)
declare noalias nonnull {} addrspace(10)* @ijl_gc_alloc_typed(i8*, i64, i8*) #4

attributes #0 = { inaccessiblemem_or_argmemonly nofree "enzyme_inactive" "enzymejl_world"="31504" }
attributes #1 = { alwaysinline nofree }
attributes #2 = { inaccessiblemem_or_argmemonly }
attributes #3 = { allocsize(2) }
attributes #4 = { allocsize(1) }
attributes #5 = { nounwind readnone }

!llvm.module.flags = !{!0, !1}
!llvm.dbg.cu = !{!2}

!0 = !{i32 2, !"Dwarf Version", i32 4}
!1 = !{i32 2, !"Debug Info Version", i32 3}
!2 = distinct !DICompileUnit(language: DW_LANG_Julia, file: !3, producer: "julia", isOptimized: true, runtimeVersion: 0, emissionKind: LineTablesOnly, nameTableKind: None)
!3 = !DIFile(filename: "julia", directory: ".")
!4 = distinct !DISubprogram(name: "fwddiffejulia_identity_1243wrap", linkageName: "fwddiffejulia_identity_1243wrap", scope: null, file: !5, type: !6, spFlags: DISPFlagLocalToUnit | DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !7)
!5 = !DIFile(filename: "operators.jl", directory: ".")
!6 = !DISubroutineType(types: !7)
!7 = !{}
!8 = !{!9, !9, i64 0}
!9 = !{!"jtbaa_gcframe", !10, i64 0}
!10 = !{!"jtbaa", !11, i64 0}
!11 = !{!"jtbaa"}
!12 = !{!13}
!13 = distinct !{!13, !14, !"primal"}
!14 = distinct !{!14, !" diff: %"}
!15 = !{!16}
!16 = distinct !{!16, !14, !"shadow_0"}
!17 = !{!18, !18, i64 0}
!18 = !{!"jtbaa_const", !10, i64 0}
!19 = !{!20}
!20 = distinct !{!20, !21, !"primal"}
!21 = distinct !{!21, !" diff: %ptls_load45"}
!22 = !{!23}
!23 = distinct !{!23, !21, !"shadow_0"}
!24 = !DILocation(line: 522, scope: !25, inlinedAt: !26)
!25 = distinct !DISubprogram(name: "identity", linkageName: "julia_identity_1243", scope: null, file: !5, line: 522, type: !6, scopeLine: 522, spFlags: DISPFlagDefinition | DISPFlagOptimized, unit: !2, retainedNodes: !7)
!26 = distinct !DILocation(line: 0, scope: !4)
"""

function move_to_threadsafe(ir)
    LLVM.verify(ir) # try to catch broken modules

    # So 1. serialize the module
    buf = convert(MemoryBuffer, ir)

    # 2. deserialize and wrap by a ThreadSafeModule
    return ThreadSafeContext() do ctx
        mod = parse(LLVM.Module, buf)
        ThreadSafeModule(mod)
    end
end

optlevel = LLVM.API.LLVMCodeGenLevelNone

tempTM = LLVM.JITTargetMachine(LLVM.triple(), cpu_name(), cpu_features(); optlevel)
LLVM.asm_verbosity!(tempTM, true)

lljit = JuliaOJIT()

jd_main = JITDylib(lljit)

prefix = LLVM.get_prefix(lljit)
dg = LLVM.CreateDynamicLibrarySearchGeneratorForProcess(prefix)
LLVM.add!(jd_main, dg)

function absolute_symbol_materialization(name, ptr)
    address = LLVM.API.LLVMOrcJITTargetAddress(reinterpret(UInt, ptr))
    flags = LLVM.API.LLVMJITSymbolFlags(LLVM.API.LLVMJITSymbolGenericFlagsExported, 0)
    symbol = LLVM.API.LLVMJITEvaluatedSymbol(address, flags)
    gv = if LLVM.version() >= v"15"
        LLVM.API.LLVMOrcCSymbolMapPair(name, symbol)
    else
        LLVM.API.LLVMJITCSymbolMapPair(name, symbol)
    end
    return LLVM.absolute_symbols(Ref(gv))
end

function define_absolute_symbol(jd, name)
    ptr = LLVM.find_symbol(name)
    if ptr !== C_NULL
        LLVM.define(jd, absolute_symbol_materialization(name, ptr))
        return true
    end
    return false
end

if Sys.iswindows() && Int === Int64
    # TODO can we check isGNU?
    define_absolute_symbol(jd_main, mangle(lljit, "___chkstk_ms"))
end

es = ExecutionSession(lljit)
lctm = LLVM.LocalLazyCallThroughManager(triple(lljit), es)
ism = LLVM.LocalIndirectStubsManager(triple(lljit))

activate(LLVM.Context())

mod = parse(LLVM.Module, modstr)
adjoint_name = "fwddiffejulia_identity_1243wrap"

jd = LLVM.JITDylib(lljit)

tsm = move_to_threadsafe(mod)
LLVM.add!(lljit, jd, tsm)

LLVM.lookup(lljit, adjoint_name)
wsmoses commented 6 months ago

Issue on LLVM.jl here: https://github.com/maleadt/LLVM.jl/issues/393

ChrisRackauckas commented 6 months ago

Thanks for taking the time to track it down!

leerosenthalj commented 6 months ago

Yes, thank you so much!On Mar 13, 2024, at 6:57 PM, Christopher Rackauckas @.***> wrote: Thanks for taking the time to track it down!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

wsmoses commented 6 months ago

This PR https://github.com/EnzymeAD/Enzyme.jl/pull/1342 hopefully should provide a workaround, which I've also just backported to 0.11 [and hopefully julia will register shortly].

Unfortuantely the way the backport works is to bypass the Julia Ojit @gbaraldi introduced in 1.10, so no nice backtraces/etc. But at least it won't lock up....

Closing for now (and the ojit can get reenabled once the issue on llvm is resolved)