`curl_easy_setopt: 4` when using Pkg

[alex180500]

In windows with a fresh install of juliaup with Julia 1.10.3 when using any command in the Pkg REPL gives out this message:

┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.3+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50

Everything is working correctly but there is this error continuously

[StefanKarpinski]

That error code indicates CURLE_NOT_BUILT_IN. Do you have http_proxy or https_proxy environment variables set? Or if not, do you have any git configs set to override protocols?

[alex180500]

By doing gci env:* | sort-object name in powershell I cannot see any http_proxy environment. My git config files are (git config --list):

diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt
core.autocrlf=true
core.fscache=true
core.symlinks=false
core.editor=notepad
pull.rebase=false
credential.helper=manager
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master
user.email=REDACTED
user.signingkey=REDACTED
commit.gpgsign=true
tag.gpgsign=true
gpg.program=C:\Program Files (x86)\GnuPG\bin\gpg.exe

Also by doing

using Libdl
filter!(contains("curl"), dllist())

I get this:

"C:\\Users\\aless\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\libcurl-4.dll"

[StefanKarpinski]

libcurl-4.dll seems likely to be the correct version of the library since it's the one we ship and this seems to be working for most people. Can you try this in the REPL:

using Downloads
Downloads.Curl.CURL_VERSION_STR
Downloads.download("https://pkg.julialang.org/registries", verbose=true)

[alex180500]

First of all sorry if I made a mistake on opening too many issues. Then thanks for the help. This is the output from the REPL.

julia> using Downloads

julia> Downloads.Curl.CURL_VERSION_STR
"libcurl/8.4.0 Schannel zlib/1.2.13 libssh2/1.11.0 nghttp2/1.52.0"

julia> Downloads.download("https://pkg.julialang.org/registries", verbose=true)
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.3+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50
* WARNING: failed to open cookie file ""
* Couldn't find host pkg.julialang.org in the .netrc file; using defaults
* Hostname pkg.julialang.org was found in DNS cache
*   Trying [2a04:4e42::729]:443...
* Connected to pkg.julialang.org (2a04:4e42::729) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /registries HTTP/1.1
Host: pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

* schannel: server closed the connection
< HTTP/1.1 301 EU internal redirect trigger
< Connection: close
< Content-Length: 0
< Server: Varnish
< Retry-After: 0
< Location: https://eu-central.pkg.julialang.org/registries
< x-geo-continent: EU
< x-geo-country: IT
< x-geo-region: GO
< Accept-Ranges: bytes
< Date: Wed, 29 May 2024 22:55:41 GMT
< Via: 1.1 varnish
< X-Served-By: cache-mxp6960-MXP
< X-Cache: HIT
< X-Cache-Hits: 0
< X-Timer: S1717023341.111815,VS0,VE0
<
* Closing connection
* schannel: shutting down SSL/TLS connection with pkg.julialang.org port 443
* Issue another request to this URL: 'https://eu-central.pkg.julialang.org/registries'
* Couldn't find host eu-central.pkg.julialang.org in the .netrc file; using defaults
* Found bundle for host: 0x185fba58320 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host eu-central.pkg.julialang.org
> GET /registries HTTP/1.1
Host: eu-central.pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

< HTTP/1.1 302 Moved Temporarily
< Server: nginx/1.26.0
< Date: Wed, 29 May 2024 22:55:41 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Location: /registries.conservative
< X-lb-strategy: pkgservers_roundrobin
<
* Ignoring the response-body
* Connection #1 to host eu-central.pkg.julialang.org left intact
* Issue another request to this URL: 'https://eu-central.pkg.julialang.org/registries.conservative'
* Couldn't find host eu-central.pkg.julialang.org in the .netrc file; using defaults
* Found bundle for host: 0x185fba58320 [serially]
* Can not multiplex, even if we wanted to
* Re-using existing connection with host eu-central.pkg.julialang.org
> GET /registries.conservative HTTP/1.1
Host: eu-central.pkg.julialang.org
Accept: */*
User-Agent: curl/8.4.0 julia/1.10

< HTTP/1.1 200 OK
< Server: nginx/1.26.0
< Date: Wed, 29 May 2024 22:55:41 GMT
< Content-Type: application/octet-stream
< Content-Length: 88
< Connection: keep-alive
< Last-Modified: Wed, 29 May 2024 22:43:47 GMT
< ETag: "6657afa3-58"
< Accept-Ranges: bytes
< X-lb-strategy: pkgservers_roundrobin
<
* Connection #1 to host eu-central.pkg.julialang.org left intact
"C:\\Users\\aless\\AppData\\Local\\Temp\\jl_TcXmhkFutW"

[alex180500]

Any news? issue is still ongoing, I want to help but I cannot figure out what is the problem...

[StefanKarpinski]

So I have no idea what is happening here. You seem to have the correct version of libcurl that we ship. Everything seems to be working correctly as well. But you're getting those error messages indicating that a curl option is being set that libcurl doesn't understand. I'll have to think about what to do next to debug this...

[StefanKarpinski]

Ok, please try this:

using Downloads
@eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
     puts("setopt: $option = $value")
     @check curl_easy_setopt(easy.handle, option, value)
end
Downloads.download("https://pkg.julialang.org/registries")

This monkey-patches the definition of Downloads.Curl.setopt to print out the option and value when it gets called. I'm trying to narrow down which option is triggering the warning.

[alex180500]

This is the result of the script provided (thank you by the way!!!!)

setopt: 10103 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 10010 = Ptr{UInt8} @0x0000024d0ab88818
setopt: 20312 = Ptr{Nothing} @0x0000024d4eaf39e0
setopt: 10313 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20079 = Ptr{Nothing} @0x0000024d4eaf3c80
setopt: 10029 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20011 = Ptr{Nothing} @0x0000024d4eaf3ee0
setopt: 10001 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 20219 = Ptr{Nothing} @0x0000024d4eaf4140
setopt: 10057 = Ptr{Nothing} @0x0000024d0aef8320
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\aless\.ssh\id_rsa
setopt: 10152 = C:\Users\aless\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\aless\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x0000024d7d7844a0
setopt: 44 = false
"C:\\Users\\aless\\AppData\\Local\\Temp\\jl_AhgFAPMHkN"

[giordano]

@StefanKarpinski side note, maybe setopt could print those things with @debug macros?

[StefanKarpinski]

Does it not show the error anymore when you do that? I was hoping for a setopt message right before the error and then we could infer that the option printed immediately prior was the one it's complaining about.

[alex180500]

I just noted one thing, if this code here:

using Downloads
@eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
     puts("setopt: $option = $value")
     @check curl_easy_setopt(easy.handle, option, value)
end
Downloads.download("https://pkg.julialang.org/registries")

is run from either the julia REPL or in the terminal by calling julia file.jl I get in the output this thing over here

setopt: 10103 = Ptr{Nothing} @0x0000021827b24240
setopt: 10010 = Ptr{UInt8} @0x0000021827c37c98
setopt: 20312 = Ptr{Nothing} @0x00000218294d5600
setopt: 10313 = Ptr{Nothing} @0x0000021827b24240
setopt: 20079 = Ptr{Nothing} @0x00000218294d58a0
setopt: 10029 = Ptr{Nothing} @0x0000021827b24240
setopt: 20011 = Ptr{Nothing} @0x00000218294d5b00
setopt: 10001 = Ptr{Nothing} @0x0000021827b24240
setopt: 20219 = Ptr{Nothing} @0x00000218294d5d60
setopt: 10057 = Ptr{Nothing} @0x0000021827b24240
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\aless\.ssh\id_rsa
setopt: 10152 = C:\Users\aless\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\aless\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x00000218524b46d0
setopt: 44 = false
setopt: 10097 = C:\Users\aless\miniforge3\envs\quarto\Library\ssl\certs
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\aless\.julia\juliaup\julia-1.10.4+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50

And the error is present.

But if I run from VS Code using the Julia: Execute active File in REPL I get no error. Actually, I just need to start the repl in VS Code and I can just do ]update and I also get no error.

Is there something related with Powershell? I'm using Powershell 7.4.2 in Windows Terminal 1.20.11381.0. I will update now to Powershell 7.4.3 so I'll update soon.

[StefanKarpinski]

Do you have one of the following environment variables set?

• SSL_CERT_FILE
• SSL_CERT_DIR
• JULIA_SSL_CA_ROOTS_PATH

[visr]

Yeah this error can be reduced on Windows to setting SSL_CERT_DIR to an empty folder.

mkdir empty-certs
$Env:SSL_CERT_DIR = "empty-certs"
julia

The Conda activation script of OpenSSL started setting SSL_CERT_FILE and SSL_CERT_DIR on Windows since https://github.com/conda-forge/openssl-feedstock/pull/157. So if you start Julia from an activated Conda environment with OpenSSL on Windows, you will run into this. And having OpenSSL in your Conda environment is likely, since also Python depends on it.

The easiest way to reproduce is by installing pixi, go to an empty folder and run:

D:\temp\act
❯ pixi init
✔ Initialized project in D:\temp\act\.

D:\temp\act
❯ pixi add openssl
✔ Added openssl

D:\temp\act
❯ pixi run julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.4 (2024-06-04)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> ENV["SSL_CERT_FILE"]
"D:\\temp\\act\\.pixi\\envs\\default\\Library\\ssl\\cacert.pem"

julia> ENV["SSL_CERT_DIR"]
"D:\\temp\\act\\.pixi\\envs\\default\\Library\\ssl\\certs"

julia> using Downloads

julia> @eval Downloads.Curl function setopt(easy::Easy, option::Integer, value)
            puts("setopt: $option = $value")
            @check curl_easy_setopt(easy.handle, option, value)
       end
setopt (generic function with 2 methods)

julia> Downloads.download("https://pkg.julialang.org/registries")
setopt: 10103 = Ptr{Nothing} @0x000001e33a183030
setopt: 10010 = Ptr{UInt8} @0x000001e33a11fde8
setopt: 20312 = Ptr{Nothing} @0x000001e33c8e3790
setopt: 10313 = Ptr{Nothing} @0x000001e33a183030
setopt: 20079 = Ptr{Nothing} @0x000001e33c8e3a10
setopt: 10029 = Ptr{Nothing} @0x000001e33a183030
setopt: 20011 = Ptr{Nothing} @0x000001e33c8e3c60
setopt: 10001 = Ptr{Nothing} @0x000001e33a183030
setopt: 20219 = Ptr{Nothing} @0x000001e33c8e3eb0
setopt: 10057 = Ptr{Nothing} @0x000001e33a183030
setopt: 99 = true
setopt: 52 = true
setopt: 68 = 50
setopt: 161 = 7
setopt: 10018 = curl/8.4.0 julia/1.10
setopt: 51 = 1
setopt: 10031 =
setopt: 216 = 8
setopt: 78 = 30
setopt: 20 = 20
setopt: 19 = 1
setopt: 10153 = C:\Users\visser_mn\.ssh\id_rsa
setopt: 10152 = C:\Users\visser_mn\.ssh\id_rsa.pub
setopt: 10026 = Ptr{Nothing} @0x0000000000000000
setopt: 10002 = https://pkg.julialang.org/registries
setopt: 64 = true
setopt: 10183 = C:\Users\visser_mn\.ssh\known_hosts
setopt: 13 = 0
setopt: 41 = false
setopt: 20094 = Ptr{Nothing} @0x0000000000000000
setopt: 10095 = Ptr{Nothing} @0x0000000000000000
setopt: 10023 = Ptr{Downloads.Curl.curl_slist_t} @0x000001e3454d8cc0
setopt: 44 = false
setopt: 10097 = D:\temp\act\.pixi\envs\default\Library\ssl\certs
┌ Error: curl_easy_setopt: 4
└ @ Downloads.Curl C:\Users\visser_mn\.julia\juliaup\julia-1.10.4+0.x64.w64.mingw32\share\julia\stdlib\v1.10\Downloads\src\Curl\utils.jl:50
"C:\\Users\\VISSER~1\\AppData\\Local\\Temp\\jl_9ov2H0nMbM"

The pixi run julia command runs this OpenSSL Conda activation script before starting Julia, which sets the environment variables. The openssl package relies on the ca-certificates package:

❯ pixi list
Package          Version       Build        Size       Kind   Source
ca-certificates  2024.6.2      h56e8100_0   152.9 KiB  conda  ca-certificates-2024.6.2-h56e8100_0.conda
openssl          3.3.1         h2466b09_1   8 MiB      conda  openssl-3.3.1-h2466b09_1.conda
ucrt             10.0.22621.0  h57928b3_0   1.2 MiB    conda  ucrt-10.0.22621.0-h57928b3_0.tar.bz2
vc               14.3          h8a93ad2_20  17 KiB     conda  vc-14.3-h8a93ad2_20.conda
vc14_runtime     14.40.33810   ha82c5b3_20  734.3 KiB  conda  vc14_runtime-14.40.33810-ha82c5b3_20.conda

And the SSL_CERT_DIR that it sets under Library\ssl\certs only contains an empty .keep file. By modifying the activation script locally I found that setting SSL_CERT_FILE does not seem to cause an issue, but setting an empty SSL_CERT_DIR does.

[StefanKarpinski]

What do you think the best mitigation would be? Should we check if the directory is empty and not apply the option if it is?

[StefanKarpinski]

Actually, this error seems to indicate that setting the directory to anything at all might not be supported. Does it only happen when the directory is empty?

[alex180500]

I can confirm that with my install of miniforge (which is just conda + mamba) and having done conda init powershell I get the following:

> $env:SSL_CERT_DIR
C:\Users\aless\miniforge3\Library\ssl\certs

> $env:SSL_CERT_FILE
C:\Users\aless\miniforge3\Library\ssl\cacert.pem

[StefanKarpinski]

Is C:\Users\aless\miniforge3\Library\ssl\certs and empty or non-existent directory? Does the file C:\Users\aless\miniforge3\Library\ssl\cacert.pem exist?

[visr]

At least for me, the cacert.pem file exists and the certs dir is empty. If I put a random file in there, the error is the same. If I remove the dir, we get a different, hard, error:

setopt: 10065 = non-existent-certs
ERROR: RequestError: schannel: failed to open CA file 'non-existent-certs': The system cannot find the file specified. while requesting https://pkg.julialang.org/registries
Stacktrace:
  [1] (::Downloads.var"#9#18"{…})(easy::Downloads.Curl.Easy)
    @ Downloads C:\Users\visser_mn\.julia\juliaup\julia-1.11.0-rc1+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Downloads\src\Downloads.jl:413
  [2] with_handle(f::Downloads.var"#9#18"{…}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl C:\Users\visser_mn\.julia\juliaup\julia-1.11.0-rc1+0.x64.w64.mingw32\share\julia\stdlib\v1.11\Downloads\src\Curl\Curl.jl:95

What do you think the best mitigation would be? Should we check if the directory is empty and not apply the option if it is?

I don't really know, I don't fully understand the issue. The curl_easy_setopt option we set is CURLOPT_CAPATH. I see the docs say:

The CURLOPT_CAPATH function apparently does not work in Windows due to some limitation in OpenSSL.

This may explain why we get a CURLE_NOT_BUILT_IN (4) return code. So does that mean we should not set ca roots path at all on Windows? I'm no expert though.

[StefanKarpinski]

I think we may want to print a warning there if its set but we can't honor it, at least once so the user can understand why the setting isn't respected.

[visr]

It would be less scary to have 1 warn log instead of many error logs for sure. If I'm reading this correctly it's what curl tools also do. Though I don't see this warning pop up when I do a curl command.

I tried to look into why this option doesn't work on Windows, but I can just see that it's been in the docs for over a decade. The code mentions Windows as well:

https://github.com/curl/curl/blob/282b9fe8ff3d23c1c975b6e29eb3eb083fac6751/lib/setopt.c#L2161-L2173

So if this is documented not to work on Windows perhaps even a warn log is too much?

[alex180500]

Is C:\Users\aless\miniforge3\Library\ssl\certs and empty or non-existent directory? Does the file C:\Users\aless\miniforge3\Library\ssl\cacert.pem exist?

The ssl folder is structured as follows in my pc

ssl/
├─ certs/
│  ├─ .keep
├─ cacert.pem
├─ cert.pem

I also agree that maybe logging this is not necessary if it's a known windows issue

[StefanKarpinski]

It may be a known curl issue, but if someone sets one of the relevant options via NetworkOptions and we just ignore it in Downloads then I think the user deserves a warning. Otherwise they may be confused about why they're using the wrong CA carts. They don't know or care that we use libcurl.

[visr]

I don't really understand why they set SSL_CERT_DIR to an empty dir in https://github.com/conda-forge/openssl-feedstock/pull/157. Would be nice if only SSL_CERT_FILE or some other mechanism would also work for them.

[StefanKarpinski]

But it doesn't really matter does it? The fact that the directory is empty is not actually relevant here. Whatever the contents of the directory, we cannot support it, so we should warn and not try setting the option.

[visr]

No indeed not for this repo. I meant perhaps we can ask them not to set SSL_CERT_DIR. In the Anaconda managed OpenSSL recipe they did a similar PR earlier on where they only set SSL_CERT_FILE. But that's best discussed on https://github.com/conda-forge/openssl-feedstock.