NetworkOptions.jl serializes `SYSTEM_CA_ROOTS` during precompilation?

[benlorenz]

With the latest julia nightly binaries I cannot update the registry or do any git operations. Note that this does not happen if I built nightly myself. I can also work around this by explicitly setting JULIA_SSL_CA_ROOTS_PATH.

lorenz@dragon /tmp/jv $ ./julia-3fc4f6bb24/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.435 (2023-09-09)
 _/ |\__'_|_|_|\__'_|  |  Commit 3fc4f6bb243 (4 days old master)
|__/                   |

(@v1.11) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /tmp/jv/julia-3fc4f6bb24/share/julia/stdlib/v1.11/Pkg/src/Registry/Registry.jl:69
    Updating registry at `/home/datastore/lorenz/software/julia/depot/registries/General`
ERROR: GitError(Code:ERROR, Class:SSL, failed to load CA certificates: 0xffffc200 - PK - Read/write of file failed)

On my system (openSUSE) the certificates are stored in /etc/ssl/ca-bundle.pem but julia seems to assume /etc/ssl/certs/ca-certificates.crt which does not exist.

I bisected this with the available nightlies to the Downloads.jl bump in 3fc4f6bb243. I would guess that the added precompile workload for Downloads.jl (added in https://github.com/JuliaLang/Downloads.jl/pull/226) causes the value from the build system to be stored in some precompiled files:

lorenz@dragon /tmp/jv $ ./julia-3fc4f6bb24/bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.435 (2023-09-09)
 _/ |\__'_|_|_|\__'_|  |  Commit 3fc4f6bb243 (4 days old master)
|__/                   |

julia> using NetworkOptions

julia> NetworkOptions.SYSTEM_CA_ROOTS
Base.RefValue{String}("/etc/ssl/certs/ca-certificates.crt")

This path is accepted without any checks by NetworkOptions.

While on the previous commit the value is not assigned and I get the correct path:

lorenz@dragon /tmp/jv $ ./julia-8e77b63fa7/bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.434 (2023-09-08)
 _/ |\__'_|_|_|\__'_|  |  Commit 8e77b63fa76 (5 days old master)
|__/                   |

julia> using NetworkOptions

julia> NetworkOptions.SYSTEM_CA_ROOTS
Base.RefValue{String}(#undef)

julia> NetworkOptions.ca_roots()
"/etc/ssl/ca-bundle.pem"

PS: Not sure if this bug should be here, in Downloads.jl, or NetworkOptions.jl.

[DilumAluthge]

Hmmm. This should have been fixed in https://github.com/JuliaLang/julia/pull/51237.

What specific commit of Julia nightly are you using?

[benlorenz]

Hmmm. This should have been fixed in JuliaLang/julia#51237.

What specific commit of Julia nightly are you using?

This also happens on the latest nightly from today, commit 8660cddaf4e:

lorenz@dragon /tmp/jv $ ./julia-8660cddaf4/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.458 (2023-09-13)
 _/ |\__'_|_|_|\__'_|  |  Commit 8660cddaf4e (0 days old master)
|__/                   |

julia> using NetworkOptions

julia> NetworkOptions.SYSTEM_CA_ROOTS
Base.RefValue{String}("/etc/ssl/certs/ca-certificates.crt")

(@v1.11) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = RequestError: Error reading ca cert file /etc/ssl/certs/ca-certificates.crt - mbedTLS: (-0x3E00) PK - Read/write of file failed while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry /tmp/jv/julia-8660cddaf4/share/julia/stdlib/v1.11/Pkg/src/Registry/Registry.jl:69

That PR (https://github.com/JuliaLang/NetworkOptions.jl/pull/31) only addresses the BUNDLED_KNOWN_HOSTS_FILE but not the SYSTEM_CA_ROOTS?

[DilumAluthge]

Ah, okay so it sounds like we need another PR similar to https://github.com/JuliaLang/julia/pull/51237, but to handle SYSTEM_CA_ROOTS.

[benlorenz]

This error has disappeared since JuliaLang/julia#51399, I guess because these stdlibs are now precompiled differently. The fixed path is gone now:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-DEV.583 (2023-09-30)
 _/ |\__'_|_|_|\__'_|  |  Commit ca862df7bfc (0 days old master)
|__/                   |

julia> using NetworkOptions

julia> NetworkOptions.SYSTEM_CA_ROOTS
Base.RefValue{String}(#undef)

(@v1.11) pkg> up
    Updating registry at `/tmp/jntmp2/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General`
  No Changes to `/tmp/jntmp2/environments/v1.11/Project.toml`
  No Changes to `/tmp/jntmp2/environments/v1.11/Manifest.toml`

julia> NetworkOptions.SYSTEM_CA_ROOTS
Base.RefValue{String}("/etc/ssl/ca-bundle.pem")

Not sure if this issue should be closed or if the code in NetworkOptions.jl needs to be fixed independently.

[StefanKarpinski]

Seems good to make the corresponding fix even if this issue isn't currently causing problems.