Type piracy on `Base` fails incremental sysimage builds (1.9 -> 1.10 regression)

[krynju]

Let's assume you have two different packages that do the same type piracy on Base. For example Base.any(a::Float64) = "a" (as in the attached example)

While unlikely in standard situations it can happen when you artifically modify the UUID on one package to effectively have two versions of the same package in separate sysimage layers.

A simple example could be that you have DataFrames.jl in the base layer with OrderedCollections.jl (which has type piracy) and then in the next layer you add a different package that also depends on OrderedCollections.jl. Note that in this example you swap the uuids of DataFrames.jl and OrderedCollections in the base layer to allow adding any packags in the next layer without any conflicts.

This used to work in 1.9, but now in 1.10 it doesn't.

Example (also everything attached in a zip below) sysimage_issue.zip

Package A

module A

Base.any(a::Float64) = "a"

end 

Package B

module B

Base.any(a::Float64) = "a"

end 

Script

using PackageCompiler
using Pkg

layer1_name = "layer1_$(replace(string(VERSION), "."=>"-")).so"
layer2_name = "layer2_$(replace(string(VERSION), "."=>"-")).so"

Pkg.activate("A")

create_sysimage(; sysimage_path=layer1_name)

Pkg.activate("B")

create_sysimage(; base_sysimage=layer1_name, sysimage_path=layer2_name)

Runs

1.9

➜  sysimage_issue julia +1.9 -t4 script.jl
The latest version of Julia in the `1.9` channel is 1.9.4+0.x64.linux.gnu. You currently have `1.9.3+0.x64.linux.gnu` installed. Run:

  juliaup update

to install Julia 1.9.4+0.x64.linux.gnu and update the `1.9` channel to that version.
  Activating project at `~/sysimage_issue/A`
Precompiling project...
  1 dependency successfully precompiled in 1 seconds
✔ [03m:18s] PackageCompiler: compiling incremental system image
  Activating project at `~/sysimage_issue/B`
Precompiling project...
  1 dependency successfully precompiled in 1 seconds
  1 dependency had warnings during precompilation:
┌ B [52150622-2ee4-4c53-a06e-8fa0ead96c0c]
│  WARNING: Method definition any(Float64) in module A at /home/krynju/sysimage_issue/A/src/A.jl:3 overwritten in module B at /home/krynju/sysimage_issue/B/src/B.jl:3.
│    ** incremental compilation may be fatally broken for this module **
└  
✔ [02m:47s] PackageCompiler: compiling incremental system image

1.10

➜  sysimage_issue julia +1.10 -t4 script.jl
The latest version of Julia in the `1.10` channel is 1.10.1+0.x64.linux.gnu. You currently have `1.10.0+0.x64.linux.gnu` installed. Run:

  juliaup update

to install Julia 1.10.1+0.x64.linux.gnu and update the `1.10` channel to that version.
  Activating project at `~/sysimage_issue/A`
Precompiling project...
  1 dependency successfully precompiled in 1 seconds
✔ [01m:27s] PackageCompiler: compiling incremental system image
  Activating project at `~/sysimage_issue/B`
Precompiling project...
        Info Given B was explicitly requested, output will be shown live 
WARNING: Method definition any(Float64) in module A at /home/krynju/sysimage_issue/A/src/A.jl:3 overwritten in module B at /home/krynju/sysimage_issue/B/src/B.jl:3.
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
  ? B
⠇ [00m:16s] PackageCompiler: compiling incremental system imageERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
⠋ [00m:18s] PackageCompiler: compiling incremental system imageDeclaring __precompile__(false) is not allowed in files that are being precompiled.
Stacktrace:
⠙ [00m:18s] PackageCompiler: compiling incremental system image  [1] top-level scope
    @ ~/sysimage_issue/B/src/B.jl:3
⣄ [00m:18s] PackageCompiler: compiling incremental system imageinclude(mod::Module, _path::String)
    @ Base ./Base.jl:495
  [3] _require(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2008
  [4] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:1806
  [5] #invokelatest#2
    @ Base ./essentials.jl:887 [inlined]
  [6] invokelatest
    @ Base ./essentials.jl:884 [inlined]
  [7] _require_prelocked
    @ Base ./loading.jl:1799 [inlined]
  [8] _require_prelocked
    @ Base ./loading.jl:1796 [inlined]
  [9] macro expansion
    @ Base ./lock.jl:267 [inlined]
 [10] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1791
 [11] top-level scope
    @ /tmp/jl_cP8RaLONMJ:6
in expression starting at /home/krynju/sysimage_issue/B/src/B.jl:1
in expression starting at /tmp/jl_cP8RaLONMJ:6
✖ [00m:18s] PackageCompiler: compiling incremental system image
ERROR: LoadError: failed process: Process(`/home/krynju/.julia/juliaup/julia-1.10.0+0.x64.linux.gnu/bin/julia --color=yes --startup-file=no --pkgimages=no --cpu-target=native --sysimage=layer1_1-10-0.so --project=/home/krynju/sysimage_issue/B --output-o=/tmp/jl_ajSZqbarAL.o /tmp/jl_cP8RaLONMJ`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ Base ./process.jl:565 [inlined]
  [2] run(::Cmd; wait::Bool)
    @ Base ./process.jl:480
  [3] run
    @ ./process.jl:477 [inlined]
  [4] #20
    @ ~/.julia/packages/PackageCompiler/nT5sD/ext/TerminalSpinners.jl:157 [inlined]
  [5] spin(f::PackageCompiler.var"#20#22"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners ~/.julia/packages/PackageCompiler/nT5sD/ext/TerminalSpinners.jl:164
  [6] macro expansion
    @ ~/.julia/packages/PackageCompiler/nT5sD/ext/TerminalSpinners.jl:157 [inlined]
  [7] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String, incremental::Bool)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:134
  [8] create_sysimg_object_file
    @ ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:311 [inlined]
  [9] create_sysimage(packages::Nothing; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::String, julia_init_c_file::Nothing, julia_init_h_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String)
    @ PackageCompiler ~/.julia/packages/PackageCompiler/nT5sD/src/PackageCompiler.jl:628
 [10] top-level scope
    @ ~/sysimage_issue/script.jl:13
in expression starting at /home/krynju/sysimage_issue/script.jl:13

[JeffBezanson]

It looks like the issue is not type piracy but method overwriting?

[krynju]

Yes, but technically type piracy is the only(?) way to achieve this

I'm glad it's validating this and it's throwing warnings/errors, although it would be nice to somehow opt out of errors for this use case.

Currently the only workaround for this is to remove any type piracy on Base from the base sysimage layer, which turns out to be a difficult thing to do (not impossible though)

There's piracy in many major packages like:

1. OrderedCollections.jl
2. TimeZones.jl
3. HTTP.jl

Also Compat.jl is a type piracy factory when used in some Julia version / Compat.jl version configurations

[JackDunnNZ]

We also ran into this as we previously used method overwriting as an ugly hack to workaround the problem described here: https://github.com/JuliaLang/PackageCompiler.jl/issues/743#issuecomment-1889969005

In Julia versions 1.9 and earlier, during sysimg compilation we were able to overwrite _artifact_str to download all artifacts rather than just the lazy ones, so that any missing artifacts would be downloaded during julia_init. In 1.10+, that is no longer an option because of this change - obviously the previous solution was pretty hacky but at least was functional. Of course if there is any suggestion for a better solution that would be great

[KristofferC]

https://github.com/JuliaLang/julia/pull/50578 for reference when this was introduced.