Downloads EASY_HOOK regression in v1.11.0?

[GhostOfElectric]

I have this as my startup.jl file:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.

using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy.com:8080")
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
    Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
end

In 1.10.5 everything's fine:

julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b36 (2024-08-27 14:19 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
  JULIA_HISTORY = U:\julialog\julia_repl.jl
  JULIA_NUM_THREADS = 4

(@v1.10) pkg> up
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Manifest.toml`

In 1.11.0, it's not, trying to add anything that has an artifact will fail.

julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2 (2024-10-07 11:40 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
  JULIA_HISTORY = U:\julialog\julia_repl.jl
  JULIA_NUM_THREADS = 4

(@v1.11) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

(@v1.11) pkg> add DuckDB
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
   Installed DuckDB_jll ─ v1.1.0+0
   Installed DuckDB ───── v1.1.0
     Failure artifact: DuckDB
     Failure artifact: DuckDB
ERROR: Unable to automatically download/install artifact 'DuckDB' from sources listed in 'E:\Program Files\JuliaDepot\packages\DuckDB_jll\IFrj8\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
- https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz

[KristofferC]

Doing something like

Downloads.EASY_HOOK[] = (easy, info) -> begin
    println("RUNNING EASY HOOK!!!")
end

I get

(@v1.11) pkg> add Plots
RUNNING EASY HOOK!!!
    Updating registry at `~/.julia/registries/General.toml`
RUNNING EASY HOOK!!!
   Resolving package versions...
RUNNING EASY HOOK!!!
RUNNING EASY HOOK!!!

so the hook is at least running...

[GhostOfElectric]

Changed it to:

# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.

println("Defo read startup file!")
using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
    println("RUNNING EASY HOOK!!!")
end

PS E:\Program Files\julia-1.11.0\bin> ./julia
Defo read startup file!
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0 (2024-10-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> add Plots
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
   Resolving package versions...
     Failure artifact: x265
     Failure artifact: x265
ERROR: Unable to automatically download/install artifact 'x265' from sources listed in 'E:\Program Files\JuliaDepot\packages\x265_jll\9auB3\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
- https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz
    Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz

[giordano]

Diff between Downloads.jl in v1.10.5 vs v1.11.0 is https://github.com/JuliaLang/Downloads.jl/compare/ead289a7f1be7689738aaea9b12d919424a106ef...89d3c7dded535a77551e763a437a6d31e4d9bf84, it shouldn't be difficult to bisect (assuming the issue is there and not somewhere else).

[KristofferC]

Can you try load Pkg, set the hook through Pkg.Downloads and use the Pkg commands (not the REPL mode) to add the package?

[GhostOfElectric]

@KristofferC It works fine if done like that:

julia> using Pkg

julia> Downloads.DOWNLOADER[] = nothing

julia> Downloads.EASY_HOOK[] = (easy, info) -> begin
           Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy:8080")
               Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
           Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
           end
#3 (generic function with 1 method)

julia> Pkg.update()
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

julia> Pkg.add("Plots")
   Resolving package versions...
  Downloaded artifact: x265
  Downloaded artifact: GR
  Downloaded artifact: LERC
  Downloaded artifact: XZ
  Downloaded artifact: fzf

In fact, it works fine, just going:

julia> using Pkg

julia> Pkg.update()
    Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
  No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`

It's just going via the REPL where it gets upset.

[IanButterworth]

So this seems to be a require_stdlib issue, but I've not been able to replicate it

With ~/.julia/config/startup.jl or etc/julia/startup.jl containing the above test code, the easy hook is run properly after a pkg repl switch (loading Pkg via require_stdlib)

julia % julia +1.11
Defo read startup file!
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.1 (2024-10-16)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> up
RUNNING EASY HOOK!!!
    Updating registry at `~/.julia/registries/General.toml`
...

Perhaps this is a windows-specific issue.

[topolarity]

That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state

Often a stdlib ends up pre-compiled for an unrelated project/workflow, and then the pre-compiled stdlib starts to load as a duplicate version everywhere

You have to check via Base.loaded_modules_array() whether a duplicate copy was loaded, in which case I expect you'll hit this MWE even on Linux

[topolarity]

The premise of require_stdlib is that normal code does not interact with any of the side-loaded stdlibs (they operate in isolation), which this kind of "public global" is incompatible with

require_stdlib makes it illegal for stdlibs to have "public globals" like DOWNLOADER

[IanButterworth]

I very much appreciate you being enlightened about this. So removing use of "public globals" is the fix?

[topolarity]

One fix would be to move the globals in question into Base, where uniqueness is still a thing

[topolarity]

Yeah.. I'm not happy about the restrictions either TBH. I think they're a rough combo of unintuitive, hard to enforce, and lead to rare bugs (that they are rare is worse, not better IMO) but it's the situation we have right now

Moving the globals to Base should fix the issue

FWIW, I don't really know why we added this behavior to require_stdlib in the first place.

It might be worth revisiting?

[IanButterworth]

IIUC at least one of the reasons was to ensure that if the user wants to use a newer stdlib version than was bundled, say Pkg, we can still load the version that the repl expects, to avoid breakage.

It has been a bumpy road for sure..

[topolarity]

Oof. Moving these into Base is going to be non-trivial..

default_downloader! and DOWNLOADER[] both use types from Curl.jl

You can't update DOWNLOADER[] for two copies of Downloads.jl at once if they have different Downloader types...

[KristofferC]

For Pkg, if you update it, I would want the repl to be the updated one. We just have to fix the common load failures from out of date manifests.

[IanButterworth]

We should probably fork this discussion to a dedicated issue, or a call, but for completeness. require_stdlib was added in https://github.com/JuliaLang/julia/pull/53326 which focused on fixing https://github.com/JuliaLang/julia/issues/53365

[vtjnash]

That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state

This is fixed by https://github.com/JuliaLang/julia/pull/55908, which was marked for backporting to avoid this issue