search

JuliaLang / Pkg.jl

Pkg - Package manager for the Julia programming language
https://pkgdocs.julialang.org
Other
609 stars 251 forks source link

Don't factor JLLs into the resolver decision when multiple update options are possible #3898

Open IanButterworth opened 1 month ago

IanButterworth commented 1 month ago

The package is usually the user-facing version not the JLL.

In some cases new JLL versions being available can force packages to be downgraded.

For instance FFMPEG_jll v6.1.1 exists but is incompatible with VideoIO after v1.0.9 (currently), when VideoIO took FFMPEG_jll on as a direct dep (to control compat), so this happens

(Foo) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
   Installed BinaryProvider ─ v0.5.10
   Installed FFMPEG ───────── v0.2.4
   Installed FFMPEG_jll ───── v6.1.1+0
   Installed VideoIO ──────── v1.0.8
  Downloaded artifact: FFMPEG
  No Changes to `~/Documents/GitHub/Foo/Project.toml`
    Updating `~/Documents/GitHub/Foo/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
⌃ [c87230d0] ↓ FFMPEG v0.4.1 ⇒ v0.2.4
⌃ [d6d074c3] ↓ VideoIO v1.1.0 ⇒ v1.0.8
  [b22a6f82] ↑ FFMPEG_jll v4.4.4+1 ⇒ v6.1.1+0

This can't cleanly be fixed in the registry too, because before v1.0.9 VideoIO didn't have FFMPEG_jll as a direct dep, so we would have to add deps to old versions, which feels nonideal.

One strategy would be to ignore JLLs when counting the cost of different resolver update paths. The other would be some reduced weighting for JLLs.

I prefer just ignoring JLLs.

ChrisRackauckas commented 1 month ago

I just hit this and agree, the user wants a working Julia package, not the latest JLL.

IanButterworth commented 1 week ago

@carlobaldassi do you think this would be simple to implement? I tried to give it a go but couldn't figure out how to.

carlobaldassi commented 1 week ago

I'll have a look, it might be doable. It would help having a reproducible case for testing though. Also, what's the cleanest way to get the info about whether a package is a JLL into the resolver call?

IanButterworth commented 1 week ago

I think the name suffix is the defining characteristic

carlobaldassi commented 5 days ago

I started looking into this, and in particular I tried to reproduce the issue. As it turns out, I can only reproduce it if I explicitly add FFMPEG_jll, whereas if I only install FFMPEG and VideoIO I get the latest versions of the explicitly required packages and a lower version for the jll, as expected.

This is on a clean temp environment:

(jl_OSTqmb) pkg> add FFMPEG VideoIO
   Resolving package versions...
    Updating `/tmp/jl_OSTqmb/Project.toml`
  [c87230d0] + FFMPEG v0.4.1
  [d6d074c3] + VideoIO v1.1.0
    Updating `/tmp/jl_OSTqmb/Manifest.toml`
...other packages...
⌅ [b22a6f82] + FFMPEG_jll v4.4.4+1
...other packages...

Calling up doesn't do anything, then installing FFMPEG_jll also doesn't change the other packages...

(jl_OSTqmb) pkg> add FFMPEG_jll
   Resolving package versions...
    Updating `/tmp/jl_OSTqmb/Project.toml`
⌅ [b22a6f82] + FFMPEG_jll v4.4.4+1
  No Changes to `/tmp/jl_OSTqmb/Manifest.toml`

Only calling up at this point leads to downgrading VideoIO and FFMPEG:

(jl_OSTqmb) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
    Updating `/tmp/jl_OSTqmb/Project.toml`
⌃ [c87230d0] ↓ FFMPEG v0.4.1 ⇒ v0.2.4
⌃ [d6d074c3] ↓ VideoIO v1.1.0 ⇒ v1.0.8
  [b22a6f82] ↑ FFMPEG_jll v4.4.4+1 ⇒ v6.1.1+0
    Updating `/tmp/jl_OSTqmb/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
⌃ [c87230d0] ↓ FFMPEG v0.4.1 ⇒ v0.2.4
  [92933f4c] + ProgressMeter v1.10.0
⌃ [d6d074c3] ↓ VideoIO v1.1.0 ⇒ v1.0.8
  [b22a6f82] ↑ FFMPEG_jll v4.4.4+1 ⇒ v6.1.1+0
...other packages...

So the idea of "ignoring JLLs" runs against the general idea (that is currently implemented) of giving precedence to explicitly requested packaged. That is: what if I explicitly request a JLL, should its version still be ignored? It would seem a bit strange, no?

IanButterworth commented 5 days ago

When I was seeing this I didn't have FFMPEG_jll as a direct dep and I doubt the reports from the wild do either, it's not a jll people directly dep on, generally. Note from my OP No Changes to `~/Documents/GitHub/Foo/Project.toml`

(Foo) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
   Installed BinaryProvider ─ v0.5.10
   Installed FFMPEG ───────── v0.2.4
   Installed FFMPEG_jll ───── v6.1.1+0
   Installed VideoIO ──────── v1.0.8
  Downloaded artifact: FFMPEG
  No Changes to `~/Documents/GitHub/Foo/Project.toml`
    Updating `~/Documents/GitHub/Foo/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
⌃ [c87230d0] ↓ FFMPEG v0.4.1 ⇒ v0.2.4
⌃ [d6d074c3] ↓ VideoIO v1.1.0 ⇒ v1.0.8
  [b22a6f82] ↑ FFMPEG_jll v4.4.4+1 ⇒ v6.1.1+0

I'm trying to find a reproducer. I wish I'd included st..