search

FluxML / Zygote.jl

21st century AD
https://fluxml.ai/Zygote.jl/
Other
1.48k stars 210 forks source link

Unable to precompile Zygote #1435

Closed MarkBlyth closed 1 year ago

MarkBlyth commented 1 year ago

On Julia 1.9.0, Linux 5.15.114-2, Manjaro, x86_64, Zygote won't precompile. To test get a blank slate to test from, I've uninstalled Julia, removed the .julia dir, reinstalled Julia, and still ran into the same issue. Running...

]activate --temp
add Zygote

gives


(@v1.9) pkg> activate --temp
  Activating new project at `/tmp/jl_OTEg90`

(jl_OTEg90) pkg> add Zygote
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/jl_OTEg90/Project.toml`
  [e88e6eb3] + Zygote v0.6.62
    Updating `/tmp/jl_OTEg90/Manifest.toml`
  [621f4979] + AbstractFFTs v1.3.1
  [79e6a3ab] + Adapt v3.6.2
  [fa961155] + CEnum v0.4.2
  [082447d4] + ChainRules v1.51.0
  [d360d2e6] + ChainRulesCore v1.16.0
  [bbf7d656] + CommonSubexpressions v0.3.0
  [34da2185] + Compat v4.6.1
  [9a962f9c] + DataAPI v1.15.0
  [e2d170a0] + DataValueInterfaces v1.0.0
  [163ba53b] + DiffResults v1.1.0
  [b552c78f] + DiffRules v1.15.1
  [ffbed154] + DocStringExtensions v0.9.3
  [1a297f60] + FillArrays v1.2.0
  [f6369f11] + ForwardDiff v0.10.35
  [0c68f7d7] + GPUArrays v8.7.1
  [46192b85] + GPUArraysCore v0.1.5
  [7869d1d1] + IRTools v0.4.10
  [92d709cd] + IrrationalConstants v0.2.2
  [82899510] + IteratorInterfaceExtensions v1.0.0
  [692b3bcd] + JLLWrappers v1.4.1
  [929cbde3] + LLVM v5.2.0
  [2ab3a3ac] + LogExpFunctions v0.3.24
  [1914dd2f] + MacroTools v0.5.10
  [77ba4419] + NaNMath v1.0.2
  [bac558e1] + OrderedCollections v1.6.0
  [aea7be01] + PrecompileTools v1.1.2
  [21216c6a] + Preferences v1.4.0
  [c1ae055f] + RealDot v0.1.0
  [189a3867] + Reexport v1.2.2
  [ae029012] + Requires v1.3.0
  [276daf66] + SpecialFunctions v2.2.0
  [1e83bf80] + StaticArraysCore v1.4.0
  [09ab397b] + StructArrays v0.6.15
  [3783bdb8] + TableTraits v1.0.1
  [bd369af6] + Tables v1.10.1
  [e88e6eb3] + Zygote v0.6.62
  [700de1a5] + ZygoteRules v0.2.3
  [dad2f222] + LLVMExtra_jll v0.0.22+0
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics v1.9.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v1.0.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [4536629a] + OpenBLAS_jll v0.3.21+4
  [05823500] + OpenLibm_jll v0.8.1+0
  [bea87d4a] + SuiteSparse_jll v5.10.1+6
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850b90] + libblastrampoline_jll v5.7.0+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
Precompiling project...
  ✗ Zygote
  0 dependencies successfully precompiled in 20 seconds. 48 already precompiled.
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package

Then, running import Pkg; Pkg.precompile() shows

julia> import Pkg; Pkg.precompile()
Precompiling project...
  ✗ Zygote
  0 dependencies successfully precompiled in 19 seconds. 48 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]

Failed to precompile Zygote [e88e6eb3-aa80-5325-afca-941959d7151f] to "/home/mark/.julia/compiled/v1.9/Zygote/jl_nuApZB".
Illegal inttoptr
      %magicptr = ptrtoint {} addrspace(10)* %242 to i64, !dbg !26

[29076] signal (6.-6): Aborted
in expression starting at /home/mark/.julia/packages/Zygote/JeHtr/src/precompile.jl:13
unknown function (ip: 0x7feb7101b26c)
gsignal at /usr/bin/../lib/libc.so.6 (unknown line)
abort at /usr/bin/../lib/libc.so.6 (unknown line)
unknown function (ip: 0x7feb7032eea5)
unknown function (ip: 0x7feb70458a28)
_ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb70457a58)
_ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb70458298)
_ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb70455d21)
unknown function (ip: 0x7feb703c40ff)
unknown function (ip: 0x7feb703c4a9c)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb703bbcda)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb6ba0987b)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb6ba2ae8e)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS0_15SymbolStringPtrENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS_8ArrayRefIPNS0_8JITDylibEEENS_9StringRefENS0_11SymbolStateE at /usr/bin/../lib/julia/../libLLVM-15.so (unknown line)
unknown function (ip: 0x7feb703bf249)
unknown function (ip: 0x7feb703c0c59)
unknown function (ip: 0x7feb703c1a72)
jl_generate_fptr_impl at /usr/bin/../lib/julia/libjulia-codegen.so.1 (unknown line)
jl_compile_method_internal at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
ijl_apply_generic at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
#166 at ./none:0
iterate at ./generator.jl:47 [inlined]
collect_to! at ./array.jl:840
collect_to_with_first! at ./array.jl:818
unknown function (ip: 0x7feacb46cca5)
collect at ./array.jl:792
unknown function (ip: 0x7feacb467a06)
prune! at /home/mark/.julia/packages/IRTools/jcpZw/src/passes/passes.jl:181
|> at ./operators.jl:907 [inlined]
#IR#15 at /home/mark/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:151 [inlined]
IR at /home/mark/.julia/packages/IRTools/jcpZw/src/ir/wrap.jl:146 [inlined]
code_ir at /home/mark/.julia/packages/Zygote/JeHtr/src/compiler/interface.jl:398
unknown function (ip: 0x7feacb43e5d8)
unknown function (ip: 0x7feb7064491c)
unknown function (ip: 0x7feb706440fa)
unknown function (ip: 0x7feb7064521b)
unknown function (ip: 0x7feb706460fe)
unknown function (ip: 0x7feb70662572)
unknown function (ip: 0x7feb70662f89)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7feb57cf6b3e)
unknown function (ip: 0x7feb57d9d928)
unknown function (ip: 0x7feb57af60b2)
unknown function (ip: 0x7feb57af60cf)
jl_f__call_latest at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
include at /home/mark/.julia/packages/Zygote/JeHtr/src/Zygote.jl:1 [inlined]
macro expansion at /home/mark/.julia/packages/Requires/Z8rfN/src/Requires.jl:40 [inlined]
precompile at /home/mark/.julia/packages/Zygote/JeHtr/src/Zygote.jl:63
macro expansion at /home/mark/.julia/packages/PrecompileTools/0yi7r/src/workloads.jl:74 [inlined]
top-level scope at /home/mark/.julia/packages/Zygote/JeHtr/src/Zygote.jl:84
unknown function (ip: 0x7feb7066292c)
unknown function (ip: 0x7feb70662c41)
unknown function (ip: 0x7feb70662f89)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7feb57cf6b3e)
unknown function (ip: 0x7feb57d9d928)
unknown function (ip: 0x7feb5775dbc7)
unknown function (ip: 0x7feb5775ddf2)
unknown function (ip: 0x7feb7064491c)
unknown function (ip: 0x7feb706440fa)
unknown function (ip: 0x7feb7064521b)
unknown function (ip: 0x7feb706460fe)
unknown function (ip: 0x7feb70662572)
unknown function (ip: 0x7feb70662f89)
ijl_toplevel_eval_in at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7feb57cf6b3e)
unknown function (ip: 0x7feb57e18dfe)
unknown function (ip: 0x7feb57e192bf)
unknown function (ip: 0x7feb57e193c8)
unknown function (ip: 0x7feb70690c90)
jl_repl_entrypoint at /usr/bin/../lib/julia/libjulia-internal.so.1 (unknown line)
main at /usr/bin/julia (unknown line)
unknown function (ip: 0x7feb70fb584f)
__libc_start_main at /usr/bin/../lib/libc.so.6 (unknown line)
_start at /usr/bin/julia (unknown line)
Allocations: 16061611 (Pool: 16051661; Big: 9950); GC: 23
Stacktrace:
 [1] top-level scope
   @ REPL[3]:1

Any help is appreciated.

sudosays commented 1 year ago

Same issue here, but it is my first time trying to precompile the package (as a dependency for Flux.jl).

Adding my system info.

OS: Arch Linux x86_64 Linux kernel version: 6.3.8-arch1-1 Julia version: 1.9.1 glibc version: 2.37-3 (installed 2023-05-11) Zygote version: v0.6.62

ToucheSir commented 1 year ago

Do you have the default (non-functional) julia package installed or one of the correct ones recommended on https://wiki.archlinux.org/title/Julia?

sudosays commented 1 year ago

Do you have the default (non-functional) julia package installed or one of the correct ones recommended on https://wiki.archlinux.org/title/Julia?

I had the julia package installed from the main Arch package repositories.

Precompiling Zygote:

Thanks for the heads up :) I will see about making the warning on the ArchWiki page more clear for others.

@MarkBlyth: can you confirm that using the AUR or pre-built Julia binary works for you?

MarkBlyth commented 1 year ago

Thanks all, can confirm the julia-bin package works as desired. I should have known to check the Arch wiki!