Start specific Julia version if it is specified in Project.toml/Manifest.toml

[davidanthoff]

This is probably a bit speculative at this point, but it would be fairly simply to look at a Project.toml/Manifest.toml if one is specified via the --project command line arguments and see whether these specify a specific Julia version. If they do, run things in that specific version.

Of course, right now there is no official way to specify a Julia version in a Manifest.toml. For Project.toml, the situation is a bit different: at least for packages one can specify a version in the compat section. The mybinder integration for example looks for a julia entry in the compat section to decide what Julia version to use. We could do something similar, for example if there was an entry julia = "=1.3.1" in the Project.toml, we could start Julia 1.3.1.

But maybe this should also just wait until there is an official way to record Julia versions in Manifest.toml files.

[davidanthoff]

The entire handling of this from rustup is interesting: https://rust-lang.github.io/rustup/overrides.html. We might want to add the same options here as well.

[ericphanson]

Of course, right now there is no official way to specify a Julia version in a Manifest.toml

Just to say, Pkg in Julia 1.7+ already does this! See https://github.com/JuliaLang/Pkg.jl/pull/2561 and followups.

[davidanthoff]

Ah, nice!

I think the big question is how we would want this to actually behave. One option would be that we always try to use the exact Julia version that is recorded in the manifest, but maybe that is a bit too drastic, as one probably would want to normally at least use the closest minor matching version, or something like that... Not really sure what the right design here is :)

[johnnychen94]

When this is supported, I would imagine adding some alias

alias julia='juliaup --project'
alias julia-1='juliaup launch 1'

and everything just works smarter.

[wolthom]

@davidanthoff Is someone already working on this?

We briefly discussed this topic in the juliaup-dev Slack channel. There you mentioned that it would be great to offer the options (or something similar) in the rustup overrides section.

I think the implementation of a layered configuration approach itself shouldn't be too bad. Two crates I've used for such purposes before are figment and config. If you prefer fewer dependencies in this instance, it should be possible to emulate it via HashMaps as well.

Could you outline some of the already existing parts?

What I'm aware of:

• The existing manual override via + exists (this should be the highest priority, right?)
• Handling of Manifest.toml-files with a julia_version is not yet implemented
• A default channel can be set globally in the juliaup.json file
• Local channel overrides are not yet possible (right?)

Where I'm not sure:

• Is it possible to use environment variables already?
• How should a local override and a Manifest.toml entry interact? My guess would be to give the local override a higher priority but warn in case of conflicts.
• Similarly regarding environment variables
• Where / which configuration files should be considered? I remember some discussions about .julia vs .juliaup vs something else for config locations.

[davidanthoff]

No one is working on this, as far as I know. I thought about it, but that is it :)

In my mind, the override hierarchy should probably look like this, highest priority first:

1. A channel selector used on the command-line, such as julia +beta (implemented).
2. The JULIAUP_CHANNEL environment variable (not implemented).
3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).
4. The .juliaup.toml file (not implemented, could also come later).
5. The Julia version stored in the Manifest.toml (not implemented).
6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

I think generally everything except 5 would store a channel name, and therefore be valid only on a specific user machine, i.e. not info that could be checked into source control.

For 4 things are very unclear to me right now. Questions are: is this a file that would be checked into git? Or is this a local config file that we would add to .gitignore? What exactly would we actually store in there? My gut feeling is that we would just not add this in a first implementation and think more about it.

I think 3 could also come later, although that is fairly simple to figure out how it should work, so maybe we should just add it now. We would need to provide a command line interface to use it, as we never want users to manually modify the juliaup.json file.

5 is going to be tricky :) The primary problem is that Manifest.toml doesn't store a channel name, but a Julia version. In general a user can have many channels installed that all provide the same Julia version. We probably also want to consider what should happen if a Manifest.toml stores say Julia version 1.7.2, but the user has a channel with Julia 1.7.3. Should that automatically be used? Should the Manifest.toml be updated? Or offered to be updated? What should happen if a Manifest.toml has a Julia version stored and there is no channel installed locally that matches even the minor version? Prompt that offers to install? In non-interactive mode, should we error and not start? There are also questions how the default project activation interacts with this. If a user does not specific --project=. on the command line, should the Manifest.toml even be considered at all?

I think maybe a pragmatic approach would be to start with an implementation of 2 and 3, and think a bit more about the other options.

I think the implementation of a layered configuration approach itself shouldn't be too bad. Two crates I've used for such purposes before are figment and config. If you prefer fewer dependencies in this instance, it should be possible to emulate it via HashMaps as well.

So as this implementation will sit on the "hot" path of everything, and also be included in the julialauncher executable which we are trying to keep small, I would probably go for a complete "by hand" implementation here... Where we for example don't even read config files that we don't need to read etc. I don't think that would be too cumbersome?

Also CCing @StefanKarpinski and @staticfloat, in particular on the order of overrides.

[staticfloat]

1. A channel selector used on the command-line, such as julia +beta (implemented).

👍

2. The JULIAUP_CHANNEL environment variable (not implemented).

Hmmm, I can see this being useful, but only in rare situations.

3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).

I'm not sure why this would be used. Can you explain more about this?

4. The .juliaup.toml file (not implemented, could also come later).

I'm not sure why you would use this over a Project.toml or Manifest.toml approach.

5. The Julia version stored in the Manifest.toml (not implemented).

I like this, but I think it probably needs to be enabled somehow, as it's a breaking behavioral change. Perhaps julia +manifest, or perhaps an option set at install time.

6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

The thing I'm most concerned about is that a user runs julia --project foo test.jl and sees something that works (because it uses v1.7, matching the Manifest), then they run julia, then run Pkg.activate("foo"); include("test.jl") and it breaks because their default channel is v1.8, or something like that. Also note the possibility for confusion if someone has defined JULIA_PROJECT.

I think for this manifest-matching mode to not confuse users we need one of the following:

• juliaup matches automatically, but it prints out that it's matching to a particular version when it does so.
• juliaup does not match automatically, it must be manually-enabled.

[StefanKarpinski]

Given that we already potentially autoupdate versions when starting julia via juliaup, I've increasingly been wondering what the point of having specific channels installed is. What I mean is: suppose I run julia +1.3 and I haven't added the 1.3 channel. Currently I get an error. And I can go ahead and add that channel and run it. But why make me do that? Why not just do whatever is necessary to install and run the latest version of 1.3 for me? At that point the idea of having a channel installed becomes a non-concept. You either have a version or you don't and when you use a channel it either has to get a version for that or not.

Ok, how is that related? Well it somewhat simplifies the logic of deciding what version to use since you don't have to worry about what channels are installed. You're either in "offline mode" and have to use a version that is already installed, in which case you pick the best installed version based on some policy, or you're allowed to download any version you want, in which case you pick the best of all published versions also according to some policy.

Btw, I got here because I started trying to write down an preference ordering on channels and that was hard. The ordering on versions already exists, why do we have to worry about channels at all when we're trying to pick a version not a channel?

Regarding policies, it seems like you want to choose how much of the recorded version to match exactly and how much to ignore and vary freely. The default would probably be to respect the major and minor versions and ignore the patch. That would mean that in offline mode you'd look at all the installed versions with the same major/minor version and then pick the latest one. In online mode you'd do the same but pick the latest of all patch releases with the same major/minor version. You could also pick a policy that only matches the major number or matches the whole version number.

[davidanthoff]

2. The JULIAUP_CHANNEL environment variable (not implemented).

Hmmm, I can see this being useful, but only in rare situations.

I think it would make it simple to integrate with something like https://asdf-vm.com/ for those folks that use that. It also should be super easy to implement ;)

3. A directory override that is stored in the central juliaup.json config file in the Julia depot (not implemented).

I'm not sure why this would be used. Can you explain more about this?

Initially I just copied it from the rust list ;) But here is one scenario where it might be useful: say I get a project from someone that has Julia 1.3.2 stored in the Manifest.toml. I have a 64 and 32 bit version of Julia 1.3.2 installed on my system, so at that point the launcher would presumably default to the 64 bit version, but maybe I want to temporarily (for a few days) test things with a 32 bit version of Julia. But I don't want to record that choice into the Manifest.toml because I don't want to commit this to the repo, I just want to switch this one folder for a few days over to some specific Juliaup channel. I think it would essentially just be a way to locally reconfigure a given directory without putting anything into that directory that might end up in source control. How useful that would really be in the end I don't know.

4. The .juliaup.toml file (not implemented, could also come later).

I'm not sure why you would use this over a Project.toml or Manifest.toml approach.

It would essentially be another way to locally override things. Or for example make a specific choice that can't be made in the Manifest.toml, like the 32 vs 64 bit binaries. Or maybe one could also configure auto-update behavior for this specific folder or something like that. But, this is all pretty vague in my mind right now, so I would just not do anything about this right now, we can always later add additional entries in this list.

5. The Julia version stored in the Manifest.toml (not implemented).

I like this, but I think it probably needs to be enabled somehow, as it's a breaking behavioral change. Perhaps julia +manifest, or perhaps an option set at install time.

6. The default Juliaup channel, stored in the Juliaup.json config file in the Julia depot (implemented).

The thing I'm most concerned about is that a user runs julia --project foo test.jl and sees something that works (because it uses v1.7, matching the Manifest), then they run julia, then run Pkg.activate("foo"); include("test.jl") and it breaks because their default channel is v1.8, or something like that. Also note the possibility for confusion if someone has defined JULIA_PROJECT.

I think for this manifest-matching mode to not confuse users we need one of the following:

• juliaup matches automatically, but it prints out that it's matching to a particular version when it does so.
• juliaup does not match automatically, it must be manually-enabled.

Yes, that all makes sense to me. Couple random other thoughts:

• Maybe 4 (.juliaup.toml) could actually help with this, i.e. it could be a way to locally store a Julia version that will always be used, irrespective of whether one activates a project or not. But it also feels like weird duplication...
• I do know what my preferred solution would be: make JULIA_PROJECT=@. the default :) I know that there were some security concerns around that, but I think one could probably solve them in other ways. In my mind, though, that would at the end of the day be the solution that is closest to what users would expect to happen.
• I hadn't really appreciated that with the ability to change project interactively during a running Julia session, we would then always have an aspect of a project that doesn't change when the user does that interactively... I don't like that but also don't really know how one could possibly get around that.

[davidanthoff]

@StefanKarpinski ah, interesting. A couple quick reactions:

1. I think we can most definitely do way, way more auto-things. julia +1.3 could just install the 1.3 channel etc., so I think much of the behavior you describe we could get while we still maintain the notion of "installed channels", installation of channels would just happen more often automatically. In my mind we should do that no matter what.
2. So then the question is do we need the notion of installed channels at all? I think it might still be useful for things like juliaup status, or simply the command juliaup update, but I'll have to think a bit more about that...

Btw, I got here because I started trying to write down an preference ordering on channels and that was hard. The ordering on versions already exists, why do we have to worry about channels at all when we're trying to pick a version not a channel?

I know, we already have one attempt at that in the Julia extension for the notebooks. We have the same problem there: we get a Jupyter notebook that has a Julia version recorded and then need to pick the right channel for that. This is the implementation. It is really pretty ad-hoc at the moment...

One very easy out of this would be to just be way more aggressive about auto-installing the x.y.z channels.

[StefanKarpinski]

My thought on matching a recorded version, 1.2.3 for the sake of example, is:

• prefer to match major and minor version, but use the latest patch available, so 1.2.8, if that's the latest patch version of the 1.2 release
  • no warning if the patch is at least at recent as the recorded version, i.e. 1.2.x for x ≥ 3
  • warn if the patch is less than the recorded version, i.e. 1.2.x for x < 3 (might be missing bug fixes)
• if a matching minor version isn't available (e.g. we're offline), prefer:
  • the latest patch of the smallest greater minor version, i.e. 1.3.x if available (preferred to 1.4.x, etc.)
    • slightly more aggressive warning than above (should work but might break if relying on internals)
  • otherwise, the latest patch of the closest largest lesser minor version, i.e. 1.1.x (preferred to 1.0.x, etc.)
    • most aggressive warning (may not work if code relies on newer features)

[StefanKarpinski]

In the meantime, just having JULIAUP_CHANNEL to control the default channel would be really handy as that would at least allow me to use direnv to control the default for various directories and it's pretty clear what setting that should do—it's just equivalent to having set the default channel to the variable's value.

[simonbyrne]

Coming to this a bit late, but I think we should try to integrate the julia version management with the package manager as close as possible. i.e.:

1. It should respect --project and JULIA_PROJECT as per julia
2. It should use the exact julia_version in the Manifest.toml
   • print a warning if not using the latest patch version
   • support +channel overrides (but these would not change the Manifest.toml)
3. juliaup instantiate [+channel] --project=... would
   • install the correct Julia channel using the following logic a. if specified, use channel (and modify the julia_version in the Manifest.toml accordingly) b. otherwise use version from Manifest.toml if it exists (with a warning if not the most recent patch) c. otherwise use most recent [compat]
   • call julia --project=... -e 'using Pkg; Pkg.instantiate()'
4. juliaup update [+channel] --project=... would
   • install the correct Julia channel
   • call julia --project=... -e 'using Pkg; Pkg.update'

So if I have a Manifest.toml with julia_version = "1.8.3", and the most recent patch is 1.8.5 and most recent release is 1.9.2, then

juliaup instantiate --project=...

would install julia 1.8.3 and instantiate the dependencies, but print a warning suggesting I do

juliaup instantiate +1.8 --project=...

Doing this would modify the Manifest.toml to change julia_version = "1.8.5", and install julia 1.8.5 (and precompile for 1.8.5)

If I then did

juliaup update --project=...

it would install julia 1.9.2 and Pkg.update() all the packages. If I wanted to stay on julia 1.8, I would either need to set a compat limit in the Project.toml, or use

juliaup update +1.8 --project=...

[davidanthoff]

There is a lot in @simonbyrne's list that I like :) A few thoughts on some random parts of it:

print a warning if not using the latest patch version

Or we could essentially use the julia entry in the compat section of the Project.toml to decide whether we should encourage an update, that would probably give us the most flexibility.

instantiate

I generally think that instantiate should never modify the Project/Manifest, but otherwise completely on board.

I think the operation where one manually specifies a channel and tells the system to update the project to that Julia version should be resolve, right?

juliaup CMD

So, I completely agree that ideally we would have a way to issue things like update, instantiate (and I think resolve) on projects from the command line, not just from within the package or Julia REPL. But my thinking lately was that we probably shouldn't pack that into the juliaup command, but instead have a separate juliapkg command line utility that is essentially a command line wrapper for Pkg.jl. And then (in an ideal world) I think Pkg.jl would actually call juliaup internally if it needs to install a certain Juliaup channel to properly resolve a project. One problem with that idea is that older versions of Pkg.jl wouldn't handle the Julia part properly. I think in my ideal world we might have something where Pkg.jl is able to resolve things for any Julia version, and then juliapkg would always use the latest version of Pkg.jl, or something along those lines. That would then still leave a problem if a user uses Pkg.jl from within Julia from an older version of Julia, but maybe there is some way around that as well...

[davidanthoff]

Oh, and one other thing: I generally think we can add a lot of convenience via interactive prompts. Say I run julia --project=..., then I think the julialauncher should really check whether that project is fully instantiated (i.e. Julia + packages), and if not and we would end-up in an interactive Julia REPL, show a prompt that asks whether we want to instantiate. And if things are not fully instantiate, just abort right away and instruct the user to instantiate first.

[simonbyrne]

Or we could essentially use the julia entry in the compat section of the Project.toml to decide whether we should encourage an update, that would probably give us the most flexibility.

I kind of like that the Manifest always gives exactly what is specified, it would be a bit odd that julia itself is an exception to this. Although patch releases are supposed to be non-breaking, there are occasional issues (e.g. there was a change from using RPATH -> RUNPATH linking in a patch release, which caused us enough headaches that we were stuck on an old patch release for a few weeks)

[davidanthoff]

Oh, just to be clear: I definitely want us to always start with the exact Julia version in the manifest, unless someone explicitly provides a higher priority override. I don't think we should do things like "if there is a newer patch version than in the manifest, use that". I was really just thinking about an informational message, but maybe that should just be show like info about new package updates as well.

[StefanKarpinski]

One thing I will add is that when you do julia --project you are explicitly saying "I trust this code" so it would be reasonable to automatically resolve a manifest, instantiate it and install the Julia version and any package versions that are in an existing manifest. The reason not to do that by default is that someone could start a julia REPL when they happen to be in a directory that is a project and maliciously forces them to do bad things when that happens. Similarly, if someone runs a script that somehow (we don't currently have a mechanism for this) indicates that it uses a project file, we're already running that script's code, we might as well trust it fully. So while having explicit instantiate commands is fine, we may want to just automatically install and use the right Julia version and similarly automatically instantiate and run the right package versions.

[StefanKarpinski]

So the "vision" is that when using juliaup you would be able to just to julia --project and have it automatically install the right Julia version and the right package versions for you (maybe with some interactive prompting, but then again maybe not even that).

[simonbyrne]

I think the operation where one manually specifies a channel and tells the system to update the project to that Julia version should be resolve, right?

Ah, yes, that would make more sense! It would also have the advantage of not needing to modify the Manifest.toml file from juliaup (as Pkg will handle that).

[simonbyrne]

@StefanKarpinski auto-instantiate would be fantastic for scripts. Would you envision this as part of julia itself, or a juliaup-specific feature?

[StefanKarpinski]

I think it would have to be a combination: juliaup would auto-install the necessary julia version while Julia itself would auto-install the package versions. Of course the end-user shouldn't have to care what is installing what as long as it all just works. I think we're always going to have two layers of installer, the key is to make a coherent experience for the user so they don't need to think too much about those two layers.

[GunnarFarneback]

What I do today to get scripts to just work is to start with the stanza

using Pkg
Pkg.activate(@__DIR__)
Pkg.instantiate()

but this requires the right version of Julia, which I've typically enforced by mandating a docker image. Automatically getting the right version from juliaup instead of the docker image would be a huge improvement and getting rid of the stanza would be a nice bonus.

I'd also like to note that what's being discussed here may have some not entirely trivial interactions with version specific manifests if some solution for that gets implemented.

[davidanthoff]

I thought a bit more about this, and I have another idea around this. At some level this would be quite a deviation from the current Juliaup philosophy, but I think there would be ways to phase that in gently, maybe even in a non-breaking way. Haven't really figured out how, though ;)

I'll use julialauncher in the description, but of course users would just type julia instead.

Imagine a world for a moment where we didn't have the concept of Juliaup channels, nor the concept of a default Juliaup channel or any of this. Instead, when one runs julialauncher without any extra info, julialauncher will look for a project named ~/.julia/environments/default/. For a moment let's assume that this always exists and worry how we bootstrap that later. julialauncher would then read the Julia version from the Manifest.toml from that file and launch that version of Julia with that default project activated, i.e. --project=~/.julia/environments/default/. So, in this design, instead of storing the default Julia version in a Juliaup config file, we store that info in the manifest of the default project.

If a user launches julialauncher with a project specified, i.e. julialauncher --project=SOMETHING then we read the Julia version from that manifest, and launch that version of Julia with that project activated.

In this world, juliaup update would no longer exist. Instead, the package manager would be in charge of updating Julia versions in manifest files. Instead of specifying a channel, we would use compat entries in the Project.toml for Julia itself.

So, specifically, when julialauncher looks at a project to active, it will also look at the julia item in [compat] in the Project.toml for that project, and if a newer version of Julia exists that matches, it would output a message "A new version of Julia exists, to update, run pkg> update." The package manger would have to be able to handle Julia itself, but from a user's point of view if one runs pkg> update then the package manager resolves not just package versions, but also the Julia version itself. If that leads to a change in the Julia version, the package manager ends with "Please restart Julia" or something like that.

Picking a specific Julia version now essentially becomes an exercise of either pinning Julia to a specific version with the package manager, or just generally editing the compat section of the Project.toml. Maybe the package manager could gain a compat command that would make it easier to modify the compat entry for something, so that I could for example say pkg> compat julia ~1.8 and that would be equivalent of using the 1.8 channel in the current Juliaup design. We could also have some custom compat values for Julia itself, for example pkg> compat julia lts could work. EDIT: Yes, David has in the meantime discovered that pkg> compat is already a thing :)

Some other random thoughts:

• this would probably all work better if the package manager could be invoked from a shell directly, i.e. if we say had a juliapkg or juliaproj command line tool.
• I haven't thought how we would handle different platforms in this world, i.e. the choice between 32 and 64 bit versions or different libc implementations. These should obviously not be stored in the Manifest.toml. So we would need some solution for that.
• I find the current julialauncher +CHANNEL syntax super convenient. It would be nice if that could somehow be combined with this idea of having the Manifest.toml be the "truth" about version selection... This is especially true because julialauncher --project=@something feels especially verbose... Maybe there is a way that one could use the + syntax for project selection, and maybe that could even be backwards compatible with how it works today... Not sure, though.

I see two main things that this design would do for us: 1) In generally I think it is not ideal that at the moment one can change the Julia version independently of updating a Manifest.toml. It seems to me that one by definition then will be in a state where things might not compatible. I think the core idea of this proposal is that any change to the Julia version should be done holistically by also looking at the packages in a project and making sure that everything resolves properly and then modifying a Manifest.toml. Essentially the only version selector for the Julia version would always be the version specified in some Manifest.toml. 2) As a side benefit, this would also supersede this proposal. I.e. we would automatically always have an active project, and we would get rid of the problem that users add all sorts of things to a project that is always active via the stacked environment stuff.

Alright, this is obviously not a fully fleshed out proposal, but I thought I'd throw it out here and see what others think :) CC @StefanKarpinski, @KristofferC, @staticfloat, @IanButterworth and obviously everyone else who wrote in this thread before.