Preferences not copied into project that is compiled

staticfloat commented 1 year ago

When building a package that reads in preferences, PackageCompiler should write out the unified set of preferences that the to-be-compiled project can see, just as Pkg does in its sandbox function. As an example, I have created this repository which showcases how compile-time preferences (and runtime preferences) are not picked up as I would expect. Just clone the repository and run make, it will run the test.jl file, which has comments showing each test and what it expects to get from each.

I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation) and runtime preferences not changing are just due to me not understanding how the environment stack works inside of the PackageCompiler regime.

sloede commented 1 year ago

Thanks a lot for taking an interest in this! I would be happy to help working on a fix, especially for the availability of preferences at compile time. However, I'm by far not knowledgeable enough about how PackageCompiler works internally (Preferences are not even mentioned in its source code), but I would be happy to lend a hand, e.g., by testing things or helping to weed out corner cases.

sloede commented 12 months ago

As stated on Discourse, I was able to set runtime preferences by adding the respective Project.toml/LocalPreferences.toml files to PREFIX/share/julia. However, while it is functional, it is not very convenient, as one has to muck around in an install directory to change a runtime option, plus it does not allow to have different configurations live side by side (one would have to create a separate installation for each use of the compiled library that requires a different set of runtime preferences). Thus maybe it would be a good option to add the ability to either change the LOAD_PATH ad-hoc or to stack it with a user-provided one.

KristofferC commented 12 months ago

I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation)

Hm, but packages are compiled with a simple Pkg.precompile with the target project active. So that should set the compile time preferences at that time?

sloede commented 12 months ago

I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation)

Hm, but packages are compiled with a simple Pkg.precompile with the target project active. So that should set the compile time preferences at that time?

Just to be clear that we talk about the same thing, let's assume we have the following setup:

.
├── LibMWE.jl
│   ├── Project.toml
│   └── src
│       └── LibMWE.jl
├── Project.toml

I now start Julia with the active project set to . and call PackageCompiler.create_library("LibMWE.jl", "dest_dir"; kwargs...). In this case, the preferences set in LibMWE.jl/Project.toml are baked into the library, but whatever preferences are set in ./Project.toml are ignored.

However, that means I cannot just have a single copy of the LibMWE.jl folder and compile multiple libraries in parallel with different compile-time preferences set, since the preferences currently need to be set inside the package_dir folder.

Thus, my point is that it would be imho cleaner (and more convenient) if there was an option to either respect the preferences set in the environment that invokes the PackageCompiler.jl commands, or to be able to pass a custom LocalPreferences.jl that will be considered in addition to what is living the in the package_dir.

KristofferC commented 12 months ago

I understand that situation but that seems different from the one @staticfloat is describing, I think?

staticfloat commented 12 months ago

I myself am a bit confused about what's going on in @sloede's example. Preferences are applied in LOAD_PATH order; that is in the case that you are describing where you run Julia with . as the active project, preferences set in LibMWE.jl/Project.toml should have no effect! Not unless you're activating that project as part of your build, perhaps?

However, that means I cannot just have a single copy of the LibMWE.jl folder and compile multiple libraries in parallel with different compile-time preferences set, since the preferences currently need to be set inside the package_dir folder.

Okay, looking at [your testing repository[(https://github.com/sloede/package-compiler-library-with-preferences) I think I understand the source of confusion. I think it may be the case that when you call PackageCompiler.create_library(package_dir, ...), you're telling the compilation to run with package_dir set as the active project; so it naturally looks at that project's preferences. Is that correct, Kristoffer? If so, it explains why we're both confused by this behavior.

sloede commented 11 months ago

I think it may be the case that when you call PackageCompiler.create_library(package_dir, ...), you're telling the compilation to run with package_dir set as the active project; so it naturally looks at that project's preferences

I cannot be 100% sure since I am not familiar enough with the PackageCompiler.jl code, but this seems to be the case, yes. Here's what I found so far:

During create_sysimg_object_file, the code that is run during the object file generation explicitly sets the LOAD_PATH to the package_dir (here named project) and nothing else: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L312 Later on, the Julia command to generate the object file is explicitly used with the project set as the active project: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L429-L431

Together, this means that whatever is defined in any project outside the package_dir will have no bearing on what ends up in the generated object file.

If this is true, what about the following approach to allow additional preferences to be set:

Add an additional kwarg local_preferences (default: nothing) to the create_xxx functions, which accepts a path to a LocalPreferences.toml file
If local_preferences is set, in create_sysimg_object_file create a temporary directory tmp_load_dir with an empty Project.toml, and copy the LocalPreferences.toml file to tmp_load_dir
Set the LOAD_PATH to
```
copy!(LOAD_PATH, [$(repr(tmp_load_dir)), $(repr(project))]) # Only allow loading packages from current project + set user-specified preferences
```
i.e., make sure that the additional LocalPreferences.toml has higher-precedence than any preference potentially set in package_dir

AFAICT, the LOAD_PATH will not end up in the compiled sysimage (since that wouldn't make sense already for the project), so the non-existence of the temporary directory after the sysimage has been created shouldn't be an issue. It also would not change any of the existing program flow, but only modify the code in a single place to allow user-specified overrides of the package_dir's default preferences. Due to the explicit argument, it would also prevent accidental diffusion of preferences from the PackageCompiler.jl environment to the compiled sysimage.

What do you think? I'd be happy to try to come up with a PR for this approach.

staticfloat commented 11 months ago

I had a different thought; if create_library() is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.

sloede commented 11 months ago

I had a different though; if create_library() is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.

Ah, you mean a solution that would achieve what I want without requiring any changes to the PackageCompiler source code? This is madness! 😱

sloede commented 11 months ago

But on a more serious note: I am thinking about how to poke holes into that idea, but I cannot find any at the moment... so let me try to think about it some more, and maybe run some real world tests.

staticfloat commented 11 months ago

This feels similar to the issues with in-tree builds in a C/C++ project; at first you start by just putting all of your .o files in the source tree right next to the .c files, but you eventually run into problems where you want to build multiple configurations from the same source, so you find it's best to separate the build artifacts/configuration from the source itself, and you transition to out-of-tree builds, where you have multiple ./configure invocations that refer to the same source, but have separate build and cache directories. In the same way here, we have multiple top-level projects with their own preferences and whatnot set, but all referencing the same set of source packages.

sloede commented 11 months ago

if create_library() is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.

Here's a (small) hole: create_library does not accept the path to a mere project to be activated, but expects a real package. IIRC, tere are even error messages about missing a package name and UUID.

If it were just a project to create, I agree, this would be a very elegant solution and fit your out-of-source build analogy very well. However, creating a package feels very... cumbersome. Do you know why this is the case (the package requirement) and/or if there's a way around it? I suspect that it has to do with the use of some Pkg functions internally, but I am no Pkg wiz and got lost while trying to figure out why.

KristofferC commented 11 months ago

Do you know why this is the case (the package requirement) and/or if there's a way around it?

It's just explicitly checked: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L786.

For example for apps we need to know the name of the "main" project to know which functions to start running when the app is running.

For a library, perhaps it is possible to remove that restriction.

sloede commented 11 months ago

Right. For apps, the package name is used for the default executable name here: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L789-L791 and later for some precompile statements: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L806-L810

For libraries, there's a similar check for the non-nothingness of ctx.env.pkg, and then it's used as the default library name here: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L982

I am not 100% about create_app, but it seems like one could change create_library such that the check for a package is only invoked if lib_name is not given and thus no library name cna be automatically inferred. Would that be an option maybe?

sloede commented 11 months ago

I just noticed the following:

Preferences that are read at compile time when building a library with create_library are not available at runtime. I stumbled across this when trying to load_preference(Mod, "name", "default") in some dynamically loaded (through Base.include) code in a compiled library. I then verified that Base.get_preferences() returns an empty Dict.

@staticfloat is this expected behavior/a feature (asking you as the presumed Preferencer-in-Chief)?

staticfloat commented 11 months ago

Yeah, you'd need to copy those preferences over into the default project of whatever library you're using. We don't really have a good way to provide runtime preference reading to those libraries; my best advice if you're doing this right now is to do something like this at the top level of your library:

const pref_value = @load_preference("name", "default")

and then use pref_value everywhere instead of using @load_preference() again.

sloede commented 11 months ago

I have successfully put a LocalPreferences.toml in the PREFIX/share/julia folder for a library and used the preferences therein. PREFIX/share/julia is hardcoded to be the JULIA_{DEPOT,LOAD}_PATH in set_depot_load_path, which itself is called during Julia initialization from init_julia for libraries, and from main for apps.

With this in mind, I am wondering if the following could be a sane solution:

We already bundle the project file for apps and libraries, i.e., copy it to PREFIX/share/julia/Project.toml using bundle_project. Along the same line, we could implement a bundle_preferences function that creates a PREFIX/share/julia/LocalPreferences.toml file containing all preferences visible during sysimage creation, i.e., while the package/project to be compiled is the active project. To avoid surprises/bloat, we would probably filter the preferences such that only those end up in LocalPreferences.toml for which a package is in the manifest.

That way, we'd fulfill the - IMHO - reasonable expectation that all preferences that are set during app/library creation, and which are pertinent for the package/project to be compiled, are accessible at runtime as well. What do you think?

~~Note: I am reopening this issue since it was closed prematurely (by myself).~~ EDIT: I can't reopen the issue due to insufficient privileges, it would be great if someone else could do that 🙏

JuliaLang / PackageCompiler.jl

Preferences not copied into project that is compiled #840