Required TOML.jl

[ThummeTo]

Hi,

v2.1.10 seems to require TOML.jl for the target package (that is built from), see error message.

I am using create_library with the following keywords (might be an unusual configuration):

PackageCompiler.create_library(...;
                                    lib_name=...,
                                    precompile_execution_file=...,
                                    precompile_statements_file=...,
                                    incremental=false,
                                    filter_stdlibs=false,
                                    julia_init_c_file = ...,
                                    header_files = ..., 
                                    force=true,
                                    include_transitive_dependencies=true,
                                    include_lazy_artifacts=true)

---

Error:

ERROR: ArgumentError: Package TOML not found in current path.
- Run `import Pkg; Pkg.add("TOML")` to install the TOML package.
Stacktrace:
 [1] macro expansion
   @ .\loading.jl:1630 [inlined]
 [2] macro expansion
   @ .\lock.jl:267 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base .\loading.jl:1611

  LoadError: failed process: Process(`'C:\Users\...\AppData\Local\Programs\Julia-1.9.3\bin\julia.exe' -Cnative '-JC:\Users\...\AppData\Local\Programs\Julia-1.9.3\lib\julia\sys.dll' --depwarn=yes --check-bounds=yes -g1 --color=yes --startup-file=no '--project=C:\Users\...\AppData\Local\Temp\...' -e 'using TOML, Pkg
  # For each dependency pair (UUID => PackageInfo), store preferences in Dict
  prefs = Dict{String,Any}(last(dep).name => Base.get_preferences(first(dep)) for dep in Pkg.dependencies())
  # Filter out packages without preferences
  filter!(p -> !isempty(last(p)), prefs)
  TOML.print(prefs, sorted=true)
  '`, ProcessExited(1)) [1]

  Stacktrace:
    [1] pipeline_error
      @ .\process.jl:565 [inlined]
    [2] read(cmd::Cmd)
      @ Base .\process.jl:449
    [3] read
      @ .\process.jl:458 [inlined]
    [4] dump_preferences(io::IOStream, project_dir::String)
      @ PackageCompiler C:\Users\...\.julia\packages\PackageCompiler\LLlSZ\src\PackageCompiler.jl:1536
    [5] #39
      @ C:\Users\...\.julia\packages\PackageCompiler\LLlSZ\src\PackageCompiler.jl:1552 [inlined]
    [6] open(::PackageCompiler.var"#39#40"{String}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Base .\io.jl:395
    [7] open
      @ .\io.jl:392 [inlined]
    [8] bundle_preferences(ctx::Pkg.Types.Context, dest_dir::String)
      @ PackageCompiler C:\Users\...\.julia\packages\PackageCompiler\LLlSZ\src\PackageCompiler.jl:1551
    [9] create_library(package_or_project::String, dest_dir::String; lib_name::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, force::Bool, header_files::Vector{String}, julia_init_c_file::String, julia_init_h_file::String, version::Nothing, compat_level::String, cpu_target::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, include_preferences::Bool, script::Nothing)
      @ PackageCompiler C:\Users\...\.julia\packages\PackageCompiler\LLlSZ\src\PackageCompiler.jl:1044

Message is indeed related to this line.

In Julia v1.9.3 with PackageCompiler v2.1.10 only - v2.1.9 is working without issues.

Thanks in advance, ThummeTo

[sloede]

Hm, I understand the error message but not where the error is coming from. TOML is a stdlib, just like Pkg. How come then that your Julia installation (with v1.9.3, as per the error message) cannot find it?

Does it work if you execute

using TOML

in a regular REPL session?

[ThummeTo]

Yep, calling using TOML in the REPL is working as expected!

[sloede]

Hm, then I really do not understand what's going on here 🤔. Note that you can disable the inclusion of the preferences by passing include_preferences=false to create_library, which should at least allow you to continue with your workflow.

[ThummeTo]

I added the call above, maybe some unusual keyword cocktail?

[ThummeTo]

Isn't it necessary to add a dependency entry in the corresponding Project.toml? It's only a guess, because I only had a quick look on th code.

Because even if standard libraries are available by default, they need to be registered in the Project.toml of the package itself in the [deps] section - even if no [compat] entry needs to be defined (this is automatically determined by the Julia version).

[ThummeTo]

Yep, that's exactly causing the issue, if I add the following to the Project.toml of the package I am compiling, it works just fine!

[deps]
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"

This should be added automatically somehow. Unfortunately Pkg.add() can't be used to add Pkg itself I think, so this must be done externally...

[sloede]

@ThummeTo To understand better why this error is happening, could you please try the following:

1. Clone PackageCompiler.jl (PC) and dev it
2. Add the following lines of code,

   println(stderr, "Base.LOAD_PATH = ", Base.LOAD_PATH)
   println(stderr, "JULIA_LOAD_PATH = ", get(ENV, "JULIA_LOAD_PATH", nothing))

   before the using TOML, Pkg line in the command string https://github.com/JuliaLang/PackageCompiler.jl/blob/037a72e9cbb9b110f406ae533d821723268fac9d/src/PackageCompiler.jl#L1538

3. Run your previous PC workflow with the devd version of PC and report back the output.

Thanks @fredrikekre for this suggestion 🙏

[fredrikekre]

Instead of 1. and 2. you can pkg> add PackageCompiler#fe/toml-debug and then do step 3.

[ThummeTo]

Sure.

Base.LOAD_PATH = ["@", "C:\\Users\\...\\AppData\\Local\\Temp\\jl_ARo29w"]
JULIA_LOAD_PATH = @;C:\Users\...\AppData\Local\Temp\jl_ARo29w

User identifier was replaced with "..."

[ThummeTo]

Is my description understandable? Otherwise I can try to describe it better :)

The package that uses Pkg and TOML needs both entries in its Project.toml in order to use them via using inside of the pacakge, even if these standard libraries are shipped together with Julia. using Pkg is allowed in plane Julia scripts, but within packages, one need to add a corresponding [deps] entry in the Project.toml (for standard libraries too), otherwise one will get an error message that the used library is not part of the project dependencies - which (in my opinion) causes this issue - because it can be resolved by adding exactly theses [deps] entries for Pkg and TOML.

[fredrikekre]

JULIA_LOAD_PATH = @;C:\Users...\AppData\Local\Temp\jl_ARo29w

Do you know if you modify this environment variable on your system? Normally this is empty. However, this one should not let you using TOML unless you find TOML in that temp project and/or the current project.

[ThummeTo]

Thanks for the fast replies BTW!

Hm, I don't think I am (actively) modifying the environment variable... Starting with a fresh REPL I also get:

julia> println(stderr, "JULIA_LOAD_PATH = ", get(ENV, "JULIA_LOAD_PATH", nothing))
JULIA_LOAD_PATH = nothing

[ThummeTo]

I also checked both ENV variables before create_library, they are always both the same (and not nothing).

Interesting: The script is executed as part of a test routine (so like test myPackage) that activates a dedicated environment for the tests as usual - if I run the script by itself in a fresh REPL everything works perfectly... so this is because it's running from a test-environment... do you have any ideas on this?

[fredrikekre]

Okay, that information would have been helpful from the start; Pkg sets up it's own package environment with only the required deps available.

PackageCompiler needs to add something like https://github.com/JuliaLang/PackageCompiler.jl/blob/037a72e9cbb9b110f406ae533d821723268fac9d/src/PackageCompiler.jl#L276 to dump_preferences.

[ThummeTo]

Yep, I just noticed the connection, otherwise I would have reported it at the beginning ;-) (I noticed it by the fact that the CI-tests fail)

Thanks!

[sloede]

@ThummeTo it would be great if you could check if https://github.com/JuliaLang/PackageCompiler.jl/pull/878 fixes your issue

[ThummeTo]

I am checking....

[ThummeTo]

Yes, this solves the reported issue! Much appreciated, some fast solved issues!

[sloede]

Thanks for testing. May I ask you to check it once more with the current PR branch version? I modified the algorithm a little to (hopefully) make it more robust.

[ThummeTo]

Sure.

Checked with the latest changes on https://github.com/sloede/PackageCompiler.jl#msl/update-load-path-in-dump-preferences, it's still working.

[sloede]

Great, thanks a lot

[ThummeTo]

This was not part of the v2.1.12 release, was it?

[sloede]

No, not yet.

[sloede]

@ThummeTo I just updated the PR once more - would you mind checking if the current solution still works for you? If yes, we can merge and create a new release.

[ThummeTo]

@sloede checked with the latest changes on https://github.com/sloede/PackageCompiler.jl#msl/update-load-path-in-dump-preferences, it still runs like a charm :-)

[sloede]

The new release v2.1.13 fixes this.

[ThummeTo]

Thank you ver much, I can confirm again that this solves my issue and the FMIExport CI-Pipeline is running now again.