Closed staticfloat closed 11 months ago
Thanks a lot for taking an interest in this! I would be happy to help working on a fix, especially for the availability of preferences at compile time. However, I'm by far not knowledgeable enough about how PackageCompiler works internally (Preferences are not even mentioned in its source code), but I would be happy to lend a hand, e.g., by testing things or helping to weed out corner cases.
As stated on Discourse, I was able to set runtime preferences by adding the respective Project.toml
/LocalPreferences.toml
files to PREFIX/share/julia
. However, while it is functional, it is not very convenient, as one has to muck around in an install directory to change a runtime option, plus it does not allow to have different configurations live side by side (one would have to create a separate installation for each use of the compiled library that requires a different set of runtime preferences). Thus maybe it would be a good option to add the ability to either change the LOAD_PATH
ad-hoc or to stack it with a user-provided one.
I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation)
Hm, but packages are compiled with a simple Pkg.precompile
with the target project active. So that should set the compile time preferences at that time?
I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation)
Hm, but packages are compiled with a simple
Pkg.precompile
with the target project active. So that should set the compile time preferences at that time?
Just to be clear that we talk about the same thing, let's assume we have the following setup:
.
├── LibMWE.jl
│ ├── Project.toml
│ └── src
│ └── LibMWE.jl
├── Project.toml
I now start Julia with the active project set to .
and call PackageCompiler.create_library("LibMWE.jl", "dest_dir"; kwargs...)
. In this case, the preferences set in LibMWE.jl/Project.toml
are baked into the library, but whatever preferences are set in ./Project.toml
are ignored.
However, that means I cannot just have a single copy of the LibMWE.jl
folder and compile multiple libraries in parallel with different compile-time preferences set, since the preferences currently need to be set inside the package_dir
folder.
Thus, my point is that it would be imho cleaner (and more convenient) if there was an option to either respect the preferences set in the environment that invokes the PackageCompiler.jl commands, or to be able to pass a custom LocalPreferences.jl
that will be considered in addition to what is living the in the package_dir
.
I understand that situation but that seems different from the one @staticfloat is describing, I think?
I myself am a bit confused about what's going on in @sloede's example. Preferences are applied in LOAD_PATH order; that is in the case that you are describing where you run Julia with .
as the active project, preferences set in LibMWE.jl/Project.toml
should have no effect! Not unless you're activating that project as part of your build, perhaps?
However, that means I cannot just have a single copy of the LibMWE.jl folder and compile multiple libraries in parallel with different compile-time preferences set, since the preferences currently need to be set inside the package_dir folder.
Okay, looking at [your testing repository[(https://github.com/sloede/package-compiler-library-with-preferences) I think I understand the source of confusion. I think it may be the case that when you call PackageCompiler.create_library(package_dir, ...)
, you're telling the compilation to run with package_dir
set as the active project; so it naturally looks at that project's preferences. Is that correct, Kristoffer? If so, it explains why we're both confused by this behavior.
I think it may be the case that when you call
PackageCompiler.create_library(package_dir, ...)
, you're telling the compilation to run withpackage_dir
set as the active project; so it naturally looks at that project's preferences
I cannot be 100% sure since I am not familiar enough with the PackageCompiler.jl code, but this seems to be the case, yes. Here's what I found so far:
During create_sysimg_object_file
, the code that is run during the object file generation explicitly sets the LOAD_PATH
to the package_dir
(here named project
) and nothing else:
https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L312
Later on, the Julia command to generate the object file is explicitly used with the project set as the active project:
https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L429-L431
Together, this means that whatever is defined in any project outside the package_dir
will have no bearing on what ends up in the generated object file.
If this is true, what about the following approach to allow additional preferences to be set:
local_preferences
(default: nothing
) to the create_xxx
functions, which accepts a path to a LocalPreferences.toml
filelocal_preferences
is set, in create_sysimg_object_file
create a temporary directory tmp_load_dir
with an empty Project.toml
, and copy the LocalPreferences.toml
file to tmp_load_dir
LOAD_PATH
to
copy!(LOAD_PATH, [$(repr(tmp_load_dir)), $(repr(project))]) # Only allow loading packages from current project + set user-specified preferences
i.e., make sure that the additional LocalPreferences.toml
has higher-precedence than any preference potentially set in package_dir
AFAICT, the LOAD_PATH
will not end up in the compiled sysimage (since that wouldn't make sense already for the project
), so the non-existence of the temporary directory after the sysimage has been created shouldn't be an issue. It also would not change any of the existing program flow, but only modify the code in a single place to allow user-specified overrides of the package_dir
's default preferences. Due to the explicit argument, it would also prevent accidental diffusion of preferences from the PackageCompiler.jl environment to the compiled sysimage.
What do you think? I'd be happy to try to come up with a PR for this approach.
I had a different thought; if create_library()
is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.
I had a different though; if
create_library()
is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.
Ah, you mean a solution that would achieve what I want without requiring any changes to the PackageCompiler source code? This is madness! 😱
But on a more serious note: I am thinking about how to poke holes into that idea, but I cannot find any at the moment... so let me try to think about it some more, and maybe run some real world tests.
This feels similar to the issues with in-tree builds in a C/C++ project; at first you start by just putting all of your .o
files in the source tree right next to the .c
files, but you eventually run into problems where you want to build multiple configurations from the same source, so you find it's best to separate the build artifacts/configuration from the source itself, and you transition to out-of-tree builds, where you have multiple ./configure
invocations that refer to the same source, but have separate build and cache directories. In the same way here, we have multiple top-level projects with their own preferences and whatnot set, but all referencing the same set of source packages.
if
create_library()
is using a particular directory as its active project, just create a project there for each "configuration" you want, and add/dev the packages you want to compile from there.
Here's a (small) hole: create_library
does not accept the path to a mere project to be activated, but expects a real package. IIRC, tere are even error messages about missing a package name and UUID.
If it were just a project to create, I agree, this would be a very elegant solution and fit your out-of-source build analogy very well. However, creating a package feels very... cumbersome. Do you know why this is the case (the package requirement) and/or if there's a way around it? I suspect that it has to do with the use of some Pkg functions internally, but I am no Pkg wiz and got lost while trying to figure out why.
Do you know why this is the case (the package requirement) and/or if there's a way around it?
It's just explicitly checked: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L786.
For example for apps we need to know the name of the "main" project to know which functions to start running when the app is running.
For a library, perhaps it is possible to remove that restriction.
Right. For apps, the package name is used for the default executable name here: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L789-L791 and later for some precompile statements: https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L806-L810
For libraries, there's a similar check for the non-nothingness of ctx.env.pkg
, and then it's used as the default library name here:
https://github.com/JuliaLang/PackageCompiler.jl/blob/7f0607c990003099969f449c28ad2aeffd0671f4/src/PackageCompiler.jl#L982
I am not 100% about create_app
, but it seems like one could change create_library
such that the check for a package is only invoked if lib_name
is not given and thus no library name cna be automatically inferred. Would that be an option maybe?
I just noticed the following:
Preferences that are read at compile time when building a library with create_library
are not available at runtime. I stumbled across this when trying to load_preference(Mod, "name", "default")
in some dynamically loaded (through Base.include
) code in a compiled library. I then verified that Base.get_preferences()
returns an empty Dict.
@staticfloat is this expected behavior/a feature (asking you as the presumed Preferencer-in-Chief)?
Yeah, you'd need to copy those preferences over into the default project of whatever library you're using. We don't really have a good way to provide runtime preference reading to those libraries; my best advice if you're doing this right now is to do something like this at the top level of your library:
const pref_value = @load_preference("name", "default")
and then use pref_value
everywhere instead of using @load_preference()
again.
I have successfully put a LocalPreferences.toml
in the PREFIX/share/julia
folder for a library and used the preferences therein. PREFIX/share/julia
is hardcoded to be the JULIA_{DEPOT,LOAD}_PATH
in set_depot_load_path
, which itself is called during Julia initialization from init_julia
for libraries, and from main
for apps.
With this in mind, I am wondering if the following could be a sane solution:
We already bundle the project file for apps and libraries, i.e., copy it to PREFIX/share/julia/Project.toml
using bundle_project
. Along the same line, we could implement a bundle_preferences
function that creates a PREFIX/share/julia/LocalPreferences.toml
file containing all preferences visible during sysimage creation, i.e., while the package/project to be compiled is the active project. To avoid surprises/bloat, we would probably filter the preferences such that only those end up in LocalPreferences.toml
for which a package is in the manifest.
That way, we'd fulfill the - IMHO - reasonable expectation that all preferences that are set during app/library creation, and which are pertinent for the package/project to be compiled, are accessible at runtime as well. What do you think?
Note: I am reopening this issue since it was closed prematurely (by myself). EDIT: I can't reopen the issue due to insufficient privileges, it would be great if someone else could do that 🙏
When building a package that reads in preferences, PackageCompiler should write out the unified set of preferences that the to-be-compiled project can see, just as
Pkg
does in itssandbox
function. As an example, I have created this repository which showcases how compile-time preferences (and runtime preferences) are not picked up as I would expect. Just clone the repository and runmake
, it will run thetest.jl
file, which has comments showing each test and what it expects to get from each.I believe compile-time preferences are a bug in PackageCompiler (it should copy the preferences over into its internal project/load path during package compilation) and runtime preferences not changing are just due to me not understanding how the environment stack works inside of the PackageCompiler regime.