Regression in Julia 1.12: Creating fresh base sysimage (`incremental=false`) fails with `error during bootstrap`

[DilumAluthge]

The error

Here is what the error message itself looks like:

✔ [01m:33s] PackageCompiler: creating compiler .ji image (incremental=false) ⡆ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)error during bootstrap: LoadError(at "/var/folders/lp/2fm0z_wn3wqfvj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl" line 3: LoadError(at "Base.jl" line 287: ErrorException("File \"/var/folders/lp/2fm0z_wn3wqfvj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jlbuild_h.jl\" not found")))

Here is the full stacktrace (click to expand):

``` Precompiling all packages... 4 dependencies successfully precompiled in 3 seconds. 26 already precompiled. + ./julia --startup-file=no --project=../PackageCompiler.jl -e 'import Pkg; Pkg.precompile()' + ./julia --startup-file=no --project=../PackageCompiler.jl ../mwe.jl ✔ [01m:33s] PackageCompiler: creating compiler .ji image (incremental=false) ⡆ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)error during bootstrap: LoadError(at "/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl" line 3: LoadError(at "Base.jl" line 287: ErrorException("File \"/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jlbuild_h.jl\" not found"))) ⠇ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false)ijl_errorf at /mydir/bisect-pc/julia/src/rtutils.c:77 jl_file_content_as_string at /mydir/bisect-pc/julia/src/toplevel.c:1161 [inlined] ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1177 include at ./boot.jl:436 [inlined] include at ./Base.jl:24 include at ./Base.jl:26 unknown function (ip: 0x1048f805f) jl_apply at /mydir/bisect-pc/julia/src/julia.h:2185 [inlined] do_call at /mydir/bisect-pc/julia/src/interpreter.c:127 eval_stmt_value at /mydir/bisect-pc/julia/src/interpreter.c:175 eval_body at /mydir/bisect-pc/julia/src/interpreter.c:678 jl_interpret_toplevel_thunk at /mydir/bisect-pc/julia/src/interpreter.c:889 top-level scope at Base.jl:287 jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:1005 jl_eval_module_expr at /mydir/bisect-pc/julia/src/toplevel.c:215 [inlined] jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:807 jl_parse_eval_all at /mydir/bisect-pc/julia/src/toplevel.c:1132 ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1179 include at ./boot.jl:436 unknown function (ip: 0x1047b405b) jl_apply at /mydir/bisect-pc/julia/src/julia.h:2185 [inlined] do_call at /mydir/bisect-pc/julia/src/interpreter.c:127 eval_stmt_value at /mydir/bisect-pc/julia/src/interpreter.c:175 eval_body at /mydir/bisect-pc/julia/src/interpreter.c:678 jl_interpret_toplevel_thunk at /mydir/bisect-pc/julia/src/interpreter.c:889 top-level scope at /var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl:3 jl_toplevel_eval_flex at /mydir/bisect-pc/julia/src/toplevel.c:1005 jl_parse_eval_all at /mydir/bisect-pc/julia/src/toplevel.c:1132 ijl_load_ at /mydir/bisect-pc/julia/src/toplevel.c:1179 ijl_load at /mydir/bisect-pc/julia/src/toplevel.c:1192 exec_program at /mydir/bisect-pc/julia/src/jlapi.c:858 true_main at /mydir/bisect-pc/julia/src/jlapi.c:915 jl_repl_entrypoint at /mydir/bisect-pc/julia/src/jlapi.c:1059 ✖ [00m:06s] PackageCompiler: compiling fresh sysimage (incremental=false) ERROR: LoadError: failed process: Process(setenv(`/mydir/bisect-pc/julia/usr/bin/julia --color=yes --startup-file=no --pkgimages=no --cpu-target generic --sysimage=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/corecompiler.ji --output-o=/var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sys-o.a /var/folders/lp/2fm0z_wn3wqf__vj6lc1tcgh0000gq/T/jl_YBjqXM/sysimage_packagecompiler_a12bf01a-97ce-11ef-0ebb-19685e72afd2.jl`,[...environment variables...]), ProcessExited(1)) [1] Stacktrace: [1] pipeline_error @ ./process.jl:598 [inlined] [2] read(cmd::Cmd) @ Base ./process.jl:482 [3] macro expansion @ /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:270 [inlined] [4] (::PackageCompiler.var"#12#17"{String, Cmd, Vector{String}, String, String, String, String})() @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:157 [5] spin(f::PackageCompiler.var"#12#17"{String, Cmd, Vector{String}, String, String, String, String}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY}) @ PackageCompiler.TerminalSpinners /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:164 [6] macro expansion @ /mydir/bisect-pc/PackageCompiler.jl/ext/TerminalSpinners.jl:157 [inlined] [7] (::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String})() @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:256 [8] cd(f::PackageCompiler.var"#10#15"{String, Cmd, Vector{String}, String, String, String, String, String}, dir::String) @ Base.Filesystem ./file.jl:112 [9] create_fresh_base_sysimage(stdlibs::Vector{String}; cpu_target::String, sysimage_build_args::Cmd) @ PackageCompiler /mydir/bisect-pc/PackageCompiler.jl/src/PackageCompiler.jl:243 [10] top-level scope @ /mydir/bisect-pc/mwe.jl:3 [11] include @ ./Base.jl:580 [inlined] [12] exec_options(opts::Base.JLOptions) @ Base ./client.jl:324 [13] _start() @ Base ./client.jl:532 in expression starting at /mydir/bisect-pc/mwe.jl:3 ```

Here is an example CI log that shows the error: https://github.com/JuliaLang/PackageCompiler.jl/actions/runs/11573962346/job/32217198440?pr=987

MWE

Here is an MWE:

import PackageCompiler

base_sysimage = PackageCompiler.create_fresh_base_sysimage(
    String[];
    cpu_target = PackageCompiler.default_app_cpu_target(),
    sysimage_build_args = ``,
)

[DilumAluthge]

I have bisected this to https://github.com/JuliaLang/julia/commit/db687ad2d92b1b3dbb70dda03647284e7166dabd (https://github.com/JuliaLang/julia/pull/54387)

db687ad2d92b1b3dbb70dda03647284e7166dabd is the first bad commit
commit db687ad2d92b1b3dbb70dda03647284e7166dabd (HEAD)
Author: Jeff Bezanson <jeff.bezanson@gmail.com>
Date:   Wed Jun 26 16:41:10 2024 -0400

    add mechanism for configuring system image builds (#54387)

    This adds the option to pass a filename of configuration settings when
    building the Core/compiler system image (from
    `base/compiler/compiler.jl`). This makes it easier to build different
    flavors of images, for example it can replace the hack that
    PackageCompiler uses to edit the list of included stdlibs, and makes it
    easy to change knobs you might want like max_methods.

 base/Base.jl              | 26 +++++++++++++++++++++++---
 base/compiler/compiler.jl | 12 ++++++++++++
 base/compiler/types.jl    |  4 ++--
 base/cpuid.jl             |  2 +-
 base/filesystem.jl        |  2 +-
 base/libc.jl              |  2 +-
 base/libuv.jl             |  2 +-
 base/pcre.jl              |  2 +-
 base/sysimg.jl            | 40 ++++++++++++++++++++++++----------------
 9 files changed, 66 insertions(+), 26 deletions(-)

[KristofferC]

Makes sense, we should use the functionality provided in that PR instead of the current text replacement for https://github.com/JuliaLang/PackageCompiler.jl/blob/38629b05fe5ebe088fe85611f651d81d9ade25e6/src/PackageCompiler.jl#L214.

[DilumAluthge]

We can probably split this up into two parts:

1. Pass "" as the second positional argument when we create the fresh base sysimage. This argument will be picked up by Base.jl as the value for BUILDROOT. This will fix the immediate bug we're seeing in this issue, which occurs because we currently only pass a single positional argument (namely, our sysimage_packagecompiler_$(uuid1()).jl script that we want to run), and thus Base.jl incorrectly assumes that said script is the BUILDROOT argument. This fix is in #996.
2. Teach create_fresh_base_sysimage() to use the new BuildSettings mechanism (from JuliaLang/julia#54387) in Julia 1.12+, instead of the "rewrite sysimg.jl" mechanism that we use in Julia 1.11 and earlier. This feature is implemented in #997.

I imagine that #997 will require more review and iterations than #996, and plus #996 is such a tiny change that fixes the immediate bug, so I figured I'd split them up into two different PRs.