Global preferences

[aplavin]

How to make global preferences for a package? I tried putting them to ~/.julia/environments/v1.7/Project.toml, but they are only available when this environment is in the LOAD_PATH. Code running from tests or from Pluto notebooks doesn't have this load path entry, so those preferences are unavailable. I'm talking about machine-specific preferences, like some system directory path, or hardware-related settings. They seem to be a good fit for Preferences.jl, but such preferences should always be available in order to make sense at all.

[vchuravy]

Global preferences essentially work through the JULIA_LOAD_PATH. See https://juliaparallel.org/tutorials/preferences/#interactions_with_the_julia_load_path

[staticfloat]

Yeah, you'll need to provide a global entry on the LOAD_PATH for this to work.

[aplavin]

Just checking – any way to have global (machine/depot wide) preferences now? :) In some common setups, LOAD_PATH doesn't contain the default environment – eg in notebooks or in tests. While machine-specific preferences should be used no matter how the code is run. Stuff like defining library paths or hardware settings.

[carstenbauer]

Just checking – any way to have global (machine/depot wide) preferences now? :)

What I do a lot (on different HPC clusters where I want global preferences) is to save a Project.toml that has all the preferences I need on this machine at /some/path/Project.toml and then set

export JULIA_LOAD_PATH=:/some/path

Note that the colon after the equal sign is crucial as it means "append". This works very well for me. See https://juliahpc.github.io/sysadmin_julia/#providing_global_package_preferences.

[aplavin]

JULIA_LOAD_PATH and LOAD_PATH variables often don't propagate: some examples I know of are running tests (Pkg.test()) and notebooks (Pluto). So this solution isn't truly "global", it only applies in limited situations.

[carstenbauer]

That's unfortunate, indeed. Although I would argue that setting environment variables is a pretty canonical way to set global settings (well beyond Julia) and would therefore see this as more of a Pluto or Pkg.test() issue.

(And, to be fair, both Pluto and Pkg.test() are known for their "sandboxing" approach, which makes propagating global information more difficult. I would say the issue is rather specific to them.)

[aplavin]

Julia environments carry a lot more than preferences – most notably, packages. Naturally, tests and notebooks don't want other packages (not under their control) to influence the setup, so they don't propagate LOAD_PATH. I assume this is an explicit decision, not an oversight. Still, global preferences (data or library paths, hardware settings) make sense to apply there as well.

And, to be fair, both Pluto and Pkg.test() are known for their "sandboxing" approach, which makes propagating global information more difficult. I would say the issue is rather specific to them.

So there's indeed no way to set Preferences globally. Tests and notebooks are common, these aren't some niche usecases. Would be nice to have a depot-wide Preferences.toml file!

[carstenbauer]

I could also imagine a JULIA_PREFERENCES_LOAD_PATH environment variable that only loads preferences (and that defaults to the regular load path). Or alternatively, the preferences from these paths could be added on top of what LOAD_PATH already gives.

[3f6a]

JULIA_LOAD_PATH and LOAD_PATH variables often don't propagate: some examples I know of are running tests (Pkg.test()) and notebooks (Pluto). So this solution isn't truly "global", it only applies in limited situations.

I just want to say that Pluto is precisely the context where I'd like to be able to set global preferences for a computer / user account. If JULIA_LOAD_PATH and LOAD_PATH do not work for this purpose, it would be nice to have some other solution. I think it would be generally useful.

[aplavin]

Are there any issues with adding ~/.julia/Preferences.toml to the default preferences files list?

[carstenbauer]

What if my depot isn't under the default path (in home)? For example, I'm thinking of a scenario where a system admin wants to set the global preferences for all users (say, on an HPC cluster). In this case, the issue is that the sys admin doesn't control where a user puts their Julia depot. (And it is common to put the depot on a parallel filesystem, i.e. outside of home.)

Also, communicating this information through a file at a fixed place isn't optimal. It's fragile (the user might wipe the depot at some point) and non trivial (when/how should the sys admin create this file? It's much easier to set an environment variable).

If we would choose this "global fixed file" approach, I'd probably (in addition) want a Preferences.toml in a subfolder of Sys.BINDIR (c.f. startup.jl)

[aplavin]

What if my depot isn't under the default path (in home)?

Sorry for ambiguity, I meant Preferences.toml in the depot folder. Or any specific subfolder, like config that is already used for stuff like startup.jl.

It's fragile (the user might wipe the depot at some point)

And that's fine, isn't it? Wipe the depot => the entire Julia installation is removed, including its settings.

non trivial (when/how should the sys admin create this file? It's much easier to set an environment variable)

The presence of a global preferences file doesn't preclude adding env vars for preferences, or to customize the location of this file for complex setups. I'm just saying there should be a way to put the global preferences file somewhere so that it's read by default no matter how Julia is started. The vast majority of computers are effectively single-user, so "truly global" and "user-global" are typically the same.

[carstenbauer]

I think we should entangle two things here.

The first point is:

Julia environments carry a lot more than preferences – most notably, packages. Naturally, tests and notebooks don't want other packages (not under their control) to influence the setup, so they don't propagate LOAD_PATH. I assume this is an explicit decision, not an oversight. Still, global preferences (data or library paths, hardware settings) make sense to apply there as well.

I think it's fair to ask for a way to set preferences of packages without making them available to load (i.e. using). That is, I can imagine a mechanism that looks at LOAD_PATH to see which packages are available and sets preferences for those packages not just based on the environments specified in LOAD_PATH directly but also other Preferences.toml files.

I think it's fair to ask for a way to load preferences from Preferences.tomls or Project.tomls without making their [deps] (if any) available to load (i.e. using).

The second point is:

How should it be possible to specify the location of these Preferences.toml/Project.toml files that should be considered in this "preferences-only" fashion? The two proposals so far are

1) A fixed-location file approach, where a Preferences.toml can be placed at a fixed location in a Julia depot (similar to config/startup.jl). 2) An environment variable approach, where a new JULIA_PREFERENCES_PATH is used to indicate the locations.

The advantage of 1) is that it is simple. The disadvantage of 1), at least in its current form, is that it doesn't give "truly global" but only "user depot global" preferences, which can be an issue in some scenarios (e.g. multi-user server settings like HPC). However, this could be amended by including a fixed location etc/Preferences.toml in the Julia install directory (not the depot). This is similar to the global etc/startup.jl which we already support (although most people likely don't know about this).

The advantage of 2) is that it is more flexible. The disadvantage is that Pluto and Pkg.test etc. would likely need to explicitly support this new environment variable (I assume that they drop all env vars and only keep the information that they need).

[aplavin]

I think it's fair to ask for a way to set preferences of packages without making them available to load (i.e. using).

Oh, I didn't even know it's not possible :) So, setting a Preference for a package is currently not allowed without having the package installed in the same env? Sorry for these basic questions, didn't really play with preferences after I understood that there's no support for global.

[carstenbauer]

I misspoke a little bit and have updated my post above. It is possible to create a Preferences.toml that only provides preferences and not any packages (i.e. it doesn't have any [deps]). See for example https://juliahpc.github.io/sysadmin_julia/#example_mpijl_and_cudajl.

However, loading preferences and making packages loadable (i.e. using) is currently somewhat intertwined in the sense that we use LOAD_PATH to find preferences but also packages (i.e. [deps]). And I thinks it is fair to think about entangling this a little bit.

Note that a tangent to this is that it would be nice to have public API here that allows one to access available preferences for a package that is not a [deps] for any environment in LOAD_PATH. In https://github.com/JuliaGPU/CUDA_Runtime_Discovery.jl/pull/13, we use Base.get_preferences() for this but that's not public API (see the resolved discussion in this PR).