curl 8.8.0 - problem with `--write-out`: error 43 - A libcurl function was given a bad argument

luckman212 commented 3 months ago

With latest v8.8.0 version of curl I am seeing a problem when using the --write-out (-w) option to print variables such as http response codes.

curl info

$ curl -V
curl 8.8.0 (aarch64-apple-darwin23.4.0) libcurl/8.8.0 SecureTransport (OpenSSL/3.3.0) zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libssh2/1.11.0 nghttp2/1.61.0 librtmp/2.3 OpenLDAP/2.6.8
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

`brew config` AND `brew doctor` output

~ $ brew update && brew upgrade
==> Updating Homebrew...
Already up-to-date.

~ $ brew dr
Your system is ready to brew.

~ $ brew config
HOMEBREW_VERSION: 4.3.2
ORIGIN: https://github.com/Homebrew/brew
HEAD: 9fdb0d53990d3e62e25325a6f89dbb769e84a530
Last commit: 4 days ago
Core tap JSON: 31 May 04:35 UTC
Core cask tap JSON: 31 May 04:35 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_AUTO_UPDATE_SECS: 14400
HOMEBREW_BAT: set
HOMEBREW_CASK_OPTS: []
HOMEBREW_CLEANUP_MAX_AGE_DAYS: 30
HOMEBREW_EDITOR: editor_subl
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_GITHUB_PACKAGES_TOKEN: set
HOMEBREW_GITHUB_PACKAGES_USER: luckman212
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_NO_ENV_HINTS: set
HOMEBREW_NO_INSECURE_REDIRECT: set
HOMEBREW_TEMP: /private/tmp/brew
Homebrew Ruby: 3.3.1 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.1/bin/ruby
CPU: octa-core 64-bit arm_blizzard_avalanche
Clang: 15.0.0 build 1500
Git: 2.45.1 => /opt/homebrew/bin/git
Curl: 8.6.0 => /usr/bin/curl
macOS: 14.5-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4
Rosetta 2: false

Verification

[X] My brew doctor output says Your system is ready to brew. and am still able to reproduce my issue.
[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.
[X] I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

use curl to make an HTTP request to GitHub and return a response code

What happened (include all command output)?

curl failed with:

$ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
curl: (43) A libcurl function was given a bad argument
000

What did you expect to happen?

get http response code 302 same as previous version and the built in /usr/bin/curl:

$ /usr/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
302

Step-by-step reproduction instructions (by running `brew` commands)

$ brew install curl

$ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1

result:

curl: (43) A libcurl function was given a bad argument

carlocab commented 3 months ago

Can't reproduce.

❯ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
302%

❯ brew info curl
==> curl: stable 8.8.0 (bottled), HEAD [keg-only]
Get a file from an HTTP, HTTPS or FTP server
https://curl.se
Installed
/opt/homebrew/Cellar/curl/8.8.0 (527 files, 4.3MB)
  Poured from bottle on 2024-05-24 at 15:27:13
[snip]

luckman212 commented 3 months ago

Is your brew config roughly the same as mine? I've reproduced this issue on both of my Macs: an M1 Mini and M2 Air, both running 14.5

carlocab commented 3 months ago

Everything relevant seems to be the same:

HOMEBREW_VERSION: 4.3.2-34-g968d56c
ORIGIN: https://github.com/Homebrew/brew
HEAD: 968d56cb7aab8647f9c9865b78fd33d568e8b060
Last commit: 9 hours ago
Core tap HEAD: 64050a1adc98e813bd9257a59216ad7fe55b344b
Core tap last commit: 5 hours ago
Core tap JSON: 13 May 09:28 UTC
Core cask tap HEAD: ffaa69778967754b3fff8e3afbf01df5f4c38ef5
Core cask tap last commit: 4 hours ago
Core cask tap JSON: 13 May 09:26 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_AUTO_UPDATE_SECS: 43200
HOMEBREW_BAT: set
HOMEBREW_BOOTSNAP: set
HOMEBREW_CASK_OPTS: []
HOMEBREW_DEVELOPER: set
HOMEBREW_EDITOR: nvim
HOMEBREW_FORCE_BREWED_CURL: set
HOMEBREW_FORCE_BREWED_GIT: set
HOMEBREW_GIT_EMAIL: 30379873+carlocab@users.noreply.github.com
HOMEBREW_GIT_NAME: Carlo Cabrera
HOMEBREW_INSTALL_BADGE: ☕
HOMEBREW_MAKE_JOBS: 16
HOMEBREW_NO_CLEANUP_FORMULAE: zsh
HOMEBREW_NO_INSTALL_FROM_API: set
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.1 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.1/bin/ruby
CPU: 16-core 64-bit arm_palma
Clang: 15.0.0 build 1500
Git: 2.45.1 => /opt/homebrew/opt/git/bin/git
Curl: 8.8.0 => /opt/homebrew/opt/curl/bin/curl
macOS: 14.5-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4
Rosetta 2: false

gromgit commented 3 months ago

@luckman212 , what's the output of otool -L /opt/homebrew/opt/curl/bin/curl on your systems?

carlocab commented 3 months ago

Tried this on a different machine and I still can't reproduce.

❯ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
302%

HOMEBREW_VERSION: 4.3.2-25-gc8f0e04
ORIGIN: https://github.com/Homebrew/brew
HEAD: c8f0e04cd6a7d08f0626ea0dfafa77bb4a7c22a5
Last commit: 24 hours ago
Core tap HEAD: 9322387999c89d777bb44429ab9149aaa2d70201
Core tap last commit: 26 hours ago
Core tap JSON: 30 Jan 07:22 UTC
Core cask tap JSON: 30 Jan 07:22 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_BAT: set
HOMEBREW_BOOTSNAP: set
HOMEBREW_CASK_OPTS: []
HOMEBREW_COLOR: set
HOMEBREW_DEVELOPER: set
HOMEBREW_FORCE_BREWED_CURL: set
HOMEBREW_FORCE_BREWED_GIT: set
HOMEBREW_GITHUB_PACKAGES_TOKEN: set
HOMEBREW_GITHUB_PACKAGES_USER: carlocab
HOMEBREW_GIT_EMAIL: 30379873+carlocab@users.noreply.github.com
HOMEBREW_GIT_NAME: Carlo Cabrera
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_NO_AUTO_UPDATE: set
HOMEBREW_NO_CLEANUP_FORMULAE: zsh
HOMEBREW_NO_INSTALL_FROM_API: set
HOMEBREW_PRY: set
HOMEBREW_SORBET_RUNTIME: set
HOMEBREW_VERIFY_ATTESTATIONS: set
Homebrew Ruby: 3.3.1 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.1/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 15.0.0 build 1500
Git: 2.45.1 => /opt/homebrew/opt/git/bin/git
Curl: 8.8.0 => /opt/homebrew/opt/curl/bin/curl
macOS: 14.5-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.4
Rosetta 2: false

Closing this as not reproducible, but feel free to carry on the conversation here.

luckman212 commented 3 months ago

@gromgit Here's the output:

$ otool -L /opt/homebrew/opt/curl/bin/curl
/opt/homebrew/opt/curl/bin/curl:
    /opt/homebrew/Cellar/curl/8.8.0/lib/libcurl.4.dylib (compatibility version 13.0.0, current version 13.0.0)
    /opt/homebrew/opt/libnghttp2/lib/libnghttp2.14.dylib (compatibility version 43.0.0, current version 43.0.0)
    /opt/homebrew/opt/libidn2/lib/libidn2.0.dylib (compatibility version 5.0.0, current version 5.0.0)
    /opt/homebrew/opt/rtmpdump/lib/librtmp.1.dylib (compatibility version 0.0.0, current version 0.0.0)
    /opt/homebrew/opt/libssh2/lib/libssh2.1.dylib (compatibility version 2.0.0, current version 2.1.0)
    /opt/homebrew/opt/openssl@3/lib/libssl.3.dylib (compatibility version 3.0.0, current version 3.0.0)
    /opt/homebrew/opt/openssl@3/lib/libcrypto.3.dylib (compatibility version 3.0.0, current version 3.0.0)
    /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
    /opt/homebrew/opt/openldap/lib/libldap.2.dylib (compatibility version 3.0.0, current version 3.200.0)
    /opt/homebrew/opt/openldap/lib/liblber.2.dylib (compatibility version 3.0.0, current version 3.200.0)
    /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.6)
    /opt/homebrew/opt/brotli/lib/libbrotlidec.1.dylib (compatibility version 1.0.0, current version 1.1.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 61123.100.169)
    /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1300.100.9)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1226.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 2420.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

gromgit commented 3 months ago

OK, nothing In your otool output seems off (was thinking perhaps your env settings were causing curl to use a different libcurl than the Homebrew one). Try the following:

brew rm --force --ignore-dependencies curl
brew install curl
/usr/bin/curl -w '%{response_code}' https://google.com

That should get you a 301 response code.

luckman212 commented 3 months ago

@gromgit Thanks, I did what you suggested. But I don't understand... with that last command you are executing the built in macOS curl... which has nothing to do with the Homebrew version? And that version has always worked.

Also, this bug or issue is specific to certain webserver response headers I think, because with google.com the problem does not appear, however with github.com it does:

Homebrew version 8.8.0, fresh install - works with google.com

$ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
301

...But doesn't work with github

$ /opt/homebrew/opt/curl/bin/curl --verbose -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
* Host github.com:443 was resolved.
* IPv6: (none)
* IPv4: 140.82.114.3
*   Trying 140.82.114.3:443...
* Connected to github.com (140.82.114.3) port 443
* ALPN: curl offers h2,http/1.1
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: github.com
*    Subject: CN=github.com
*    Issuer: C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo ECC Domain Validation Secure Server CA
*    Version: 2
*    Serial Number: 4e:28:f7:86:b6:6c:1a:3b:94:2c:d2:c4:0e:b7:42:a5:
* Closing connection
curl: (43) A libcurl function was given a bad argument
000

Built in binary (works)

/usr/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
302

gromgit commented 3 months ago

Sorry, was distracted by another issue. All the curls should be from Homebrew, of course.

Can you try the google.com line again, but this time adding the -v flag (like for the github.com one)?

luckman212 commented 3 months ago

Sure. I trimmed away a chunk of the cert to try to keep this from being too long.

$ /opt/homebrew/opt/curl/bin/curl -v -w '%{response_code}' https://google.com
* Host google.com:443 was resolved.
* IPv6: 2607:f8b0:4006:822::200e
* IPv4: 142.251.40.206
*   Trying [2607:f8b0:4006:822::200e]:443...
* Connected to google.com (2607:f8b0:4006:822::200e) port 443
* ALPN: curl offers h2,http/1.1
* TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.google.com
*    Subject: CN=*.google.com
*    Issuer: C=US, O=Google Trust Services LLC, CN=GTS CA 1C3
*    Version: 2
*    Serial Number: 00:82:1c:19:27:8e:e1:25:0f:10:b0:b8:e6:55:d6:85:e6:
*    Signature Algorithm: sha256WithRSAEncryption
*    Start Date: 2024-05-13 06:34:53 GMT
*    Expire Date: 2024-08-05 06:34:52 GMT
*    Public Key Algorithm: ecPublicKey
*    ECC Public Key (256 bits)
*    ecPublicKey: 04:32:cb:62:6e:5f:c2:dc:39:eb:a9:d7:1a:54:d9:45:1e:4c:7d:9a:f1:41:71:a7:00:9b:32:7b:01:3b:dc:dd:b2:82:44:d4:ce:e8:cf:d4:05:24:39:a0:be:14:a2:fb:eb:99:3d:e0:e2:71:a6:47:a1:72:5a:43:05:4f:87:97:5a:
*    Signature: 2f:6e:f8:59:8a:df:1c:0d:c4:ff:af:f2:21:80:58:ab:c1:0e:7f:8f:2b:6d:6e:22:4c:f5:a1:76:25:d3:22:05:dd:bf:79:f2:ae:3f:f4:4a:cf:55:3d:ee:37:7f:ae:5d:3d:1d:4f:6a:62:a4:22:bd:6f:ec:f1:28:46:33:ef:d9:35:6f:30:ef:5e:67:92:97:0a:f9:8a:62:a2:ee:d4:ff:cf:68:24:9c:bd:0e:e8:8a:a8:62:f7:f4:39:81:06:1c:21:d6:a5:93:3d:83:bb:e6:c0:01:b7:38:ec:f4:e0:9e:5f:aa:44:ff:e8:80:1c:17:e3:d1:c6:2f:8d:7d:16:97:f5:c1:bc:a7:3f:b1:ac:8a:9b:18:d5:b4:d5:b9:ab:e8:82:22:d6:99:25:3f:80:85:bf:84:4b:65:e6:db:cb:30:63:cc:5b:a2:13:21:a2:84:28:11:3b:5e:d3:5a:d0:e3:b3:94:15:0c:e6:db:b2:3d:ab:69:8b:f3:2a:93:77:e5:b8:c4:95:b8:a1:23:0f:1d:70:d0:de:d4:72:cd:b4:d6:65:c1:5d:36:21:4f:9c:73:e6:36:a6:e3:c3:36:a8:d8:c7:27:73:7b:a5:6b:98:f8:b5:01:02:68:10:fd:2f:69:1e:82:52:36:93:90:54:ca:7a:55:17:9e:34:52:11:1d:
*    Cert: -----BEGIN CERTIFICATE-----
MIIOfTCCDWWgAwIBAgIRAIIcGSeO4SUPELC45lXWheYwDQYJKoZIhvcNAQELBQAw
RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
TEMxEzARBgNVBAMTCkdUUyBDQSAxQzMwHhcNMjQwNTEzMDYzNDUzWhcNMjQwODA1
<...snip...>
Z2xlYWRzZXJ2aWNlcy1jbi5jb22CEWdvb2dsZXZhZHMtY24uY29tghMqLmdv
* Server certificate: GTS CA 1C3
* Server certificate: GTS Root R1
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://google.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: google.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.8.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: google.com
> User-Agent: curl/8.8.0
> Accept: */*
>
* Request completely sent off
< HTTP/2 301
< location: https://www.google.com/
< content-type: text/html; charset=UTF-8
< content-security-policy-report-only: object-src 'none';base-uri 'self';script-src 'nonce-Lqq_W27Oei4N7qzvsSFf1Q' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< date: Fri, 31 May 2024 15:35:47 GMT
< expires: Sun, 30 Jun 2024 15:35:47 GMT
< cache-control: public, max-age=2592000
< server: gws
< content-length: 220
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
<
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
* Connection #0 to host google.com left intact

gromgit commented 3 months ago

Sorry, I'm out of ideas, but as neither @carlocab nor I can replicate your issue, it's likely an issue specific to your environment. You might want to report an issue upstream.

luckman212 commented 3 months ago

Please see the continued discussion over here — I think it points to this being is a Homebrew build-specific issue.

chenrui333 commented 3 months ago

I cannot reproduce from my side

$ /opt/homebrew/opt/curl/bin/curl -w '%{response_code}' https://github.com/ItzLevvie/MicrosoftTeams-msinternal/raw/master/src/Get-MicrosoftTeams.ps1
302

luckman212 commented 3 months ago

After spending several hours compiling builds with various flags, I found that as long as I had --with-secure-transport enabled, it would fail. I was scratching my head until I finally found this old bit of cobweb in my .bash_profile

# Fix for `curl: (60) SSL certificate problem: certificate has expired`
export CURL_SSL_BACKEND='secure-transport'

Commenting that out has resolved this. Sorry to all for time wasted.

carlocab commented 3 months ago

That's odd. I can reproduce this, but I don't see why it should break things. Seems like a curl bug.

Glad you were able to figure it out!

Homebrew / homebrew-core