error with Julia 1.10 when loading packages that trigger precompilation

matthias314 commented 1 year ago

Suppose I want to load a package, say TestPkg.jl with content

module TestPkg end

If the package triggers precompilation, then in IJulia with kernel Julia 1.10.0-beta1 I get

In [1]: using TestPkg
MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint})
The applicable method may be too new: running in world age 31360, while current world is 31378.

The same happens for import TestPkg.

With command-line Julia 1.10 or IJulia with kernel 1.9 I get the expected result,

In [1]: using TestPkg
[ Info: Precompiling TestPkg [top-level]

I'm using Jupyter Notebook 6.3.0 with IJulia 1.24.2.

EDIT: The problem persists with Julia 1.10.0-beta2. Also, for me it appears even for trivial packages like TestPkg above, but only with IJulia: If I create a script test.jl with the line using TestPkg, then both julia test.jl and julia -L test.jl work fine if TestPkg needs precompilation.

svilupp commented 1 year ago

+1 on 1.10 beta2! I presume it's the same root cause. It broke a lot of our reports, but I could swear it used to work on 1.10 beta1 - I need to try that.

An error occurred while executing the following cell:

using MyPkg

MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint}) The applicable method may be too new: running in world age 31373, while current world is 31593.

Closest candidates are: pipe_writer(::IJulia.IJuliaStdio) (method too new to be called from this world context.) @ IJulia ~/.julia/packages/IJulia/Vo51o/src/stdio.jl:15 pipe_writer(::Base.ProcessChain) @ Base process.jl:36 pipe_writer(::Base.Process) @ Base process.jl:22 ...

Stacktrace: [1] setup_stdio(stdio::IJulia.IJuliaStdio{Base.PipeEndpoint}, readable::Bool) @ Base ./process.jl:260 [2] setup_stdios(f::Base.var"#784#785"{Cmd}, stdios::Vector{Union{RawFD, IO}}) @ Base ./process.jl:221 [3] _spawn @ Base ./process.jl:138 [inlined] [4] open(cmds::Cmd, stdio::Base.DevNull; write::Bool, read::Bool) @ Base ./process.jl:393 [5] open (repeats 2 times) @ Base ./process.jl:383 [inlined] [6] open(::Pkg.Registry.var"#19#22"{IOBuffer, Vector{UInt8}, Dict{String, String}}, ::Cmd; kwargs::@Kwargs{}) @ Base ./process.jl:415 [7] open @ Base ./process.jl:414 [inlined] [8] uncompress_registry(tar_gz::String) @ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:261 [9] Pkg.Registry.RegistryInstance(path::String) @ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:328 [10] reachable_registries(; depots::Vector{String}) @ Pkg.Registry ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:434 [11] reachable_registries @ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Registry/registry_instance.jl:404 [inlined] [12] Context @ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/Types.jl:395 [inlined] [13] precompile(pkgs::Vector{Pkg.Types.PackageSpec}; io::IJulia.IJuliaStdio{Base.PipeEndpoint}, kwargs::@Kwargs{_from_loading::Bool}) @ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:150 [14] precompile @ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined] [15] #precompile#114 @ Pkg.API ~/.julia/juliaup/julia-1.10.0-beta2+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:146 [inlined] [16] _require(pkg::Base.PkgId, env::String) @ Base ./loading.jl:1957 [17] __require_prelocked(uuidkey::Base.PkgId, env::String) @ Base ./loading.jl:1806 [18] #invoke_in_world#3 @ Base ./essentials.jl:921 [inlined] [19] invoke_in_world @ Base ./essentials.jl:918 [inlined] [20] _require_prelocked(uuidkey::Base.PkgId, env::String) @ Base ./loading.jl:1797 [21] macro expansion @ Base ./loading.jl:1784 [inlined] [22] macro expansion @ Base ./lock.jl:267 [inlined] [23] __require(into::Module, mod::Symbol) @ Base ./loading.jl:1747 [24] #invoke_in_world#3 @ Base ./essentials.jl:921 [inlined] [25] invoke_in_world @ Base ./essentials.jl:918 [inlined] [26] require(into::Module, mod::Symbol) @ Base ./loading.jl:1740 LoadError: MethodError: no method matching pipe_writer(::IJulia.IJuliaStdio{Base.PipeEndpoint}) The applicable method may be too new: running in world age 31373, while current world is 31593.

Closest candidates are: pipe_writer(::IJulia.IJuliaStdio) (method too new to be called from this world context.) @ IJulia ~/.julia/packages/IJulia/Vo51o/src/stdio.jl:15 pipe_writer(::Base.ProcessChain)

IJulia: 1.24.2

Versioninfo:

Julia Version 1.10.0-beta2 Commit a468aa198d0 (2023-08-17 06:27 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: macOS (arm64-apple-darwin22.4.0) CPU: 8 × Apple M1 Pro WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-15.0.7 (ORCJIT, apple-m1) Threads: 1 on 6 virtual cores

svilupp commented 1 year ago

It seems to be similar to: https://github.com/JuliaLang/julia/issues/28042

I can confirm that the issue is 1.10 beta2.

I can run it fine on 1.9 and 1.10 beta1.

I haven't had much luck with a reproducible example yet; it's easy to repro with our internal packages, but not with mock ones.

svilupp commented 1 year ago

This seems to be replicable on my machine:

generate a random module, eg, ABC/ABC.jl

module ABC

greet() = print("Hello World!")

struct A1 end
struct B1 end

greet(::A1) = print("Hello World!")

x(::A1) = "x"

end # module ABC

In your script simply have:

using StatsPlots # not sure why but it helps to have some complex package there
using ABC
ABC.greet(ABC.A1)

If you first open REPL and precompile it, IJulia will work fine. If you make some changes, eg, change x(::A1) = "x" to x(::B1) = "x" (which requires precompilation), it will throw the error. If you precompile "manually" by opening a fresh session and running the code, IJulia will work fine again.

Notes:

If you have any daemon running, you need to kill it first (eg, if you use Quarto)

EDIT: sometimes the change has to be bigger (eg, introduce a new struct and a new function with a method for it) to trigger the issue

EDIT2: For whatever reason, putting this at the top solves the issue??? using Pkg; Pkg.precompile(["ABC"]; strict=true); But it works only in this toy example, not in real-world scenarios.

svilupp commented 1 year ago

Quick update:

1.9.3 works perfectly fine
1.10 beta1 consistently fails
1.10 beta2 consistently fails
1.11 main consistently fails

The only workaround so far is to add explicit precompile (using Pkg; Pkg.precompile()), but it works only for the MWE. It doesn't work in my real-world use cases, so I had to revert to 1.9

svilupp commented 11 months ago

It should be fixed by https://github.com/JuliaLang/julia/pull/51397

I believe this issue can be closed

JuliaLang / IJulia.jl

error with Julia 1.10 when loading packages that trigger precompilation #1089

An error occurred while executing the following cell:

using MyPkg