cURL built with rustls-ffi 0.10.0; 0.12.1 unable to talk to Git https endpoints #407

Closed Kangie closed 3 months ago

I did this

I have used GitHub as an example here, but GitLab and Bitbucket also failed in the same way.

(chroot) monolith /tmp # GIT_TRACE_CURL=1 GIT_CURL_VERBOSE=1 git clone https://github.com/kangie/kangie-tools.git
Cloning into 'kangie-tools'...
05:45:18.595988 http.c:843              == Info: Couldn't find host github.com in the .netrc file; using defaults
05:45:18.596557 http.c:843              == Info: Host github.com:443 was resolved.
05:45:18.596568 http.c:843              == Info: IPv6: (none)
05:45:18.596572 http.c:843              == Info: IPv4: 20.248.137.48
05:45:18.596588 http.c:843              == Info:   Trying 20.248.137.48:443...
05:45:18.647927 http.c:843              == Info: Connected to github.com (20.248.137.48) port 443
05:45:18.647980 http.c:843              == Info: ALPN: curl offers http/1.1
05:45:18.648113 http.c:843              == Info: rustls_connection wants us to write_tls.
05:45:18.648156 http.c:843              == Info: Curl_socket_check: reading would block
05:45:18.703955 http.c:843              == Info: rustls_connection wants us to read_tls.
05:45:18.704143 http.c:843              == Info: rustls_connection wants us to write_tls.
05:45:18.704183 http.c:843              == Info: rustls_connection wants us to read_tls.
05:45:18.704281 http.c:843              == Info: rustls_connection_process_new_packets: invalid peer certificate: BadSignature
05:45:18.704296 http.c:843              == Info: Closing connection
fatal: unable to access 'https://github.com/kangie/kangie-tools.git/': rustls_connection_process_new_packets: invalid peer certificate: BadSignature
(chroot) monolith /tmp # curl --version
curl 8.6.0 (x86_64-gentoo-linux-musl) libcurl/8.6.0 rustls-ffi/0.10.0/rustls/0.21.0 zlib/1.3 c-ares/1.25.0 libpsl/0.21.5
Release-Date: 2024-01-31
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp
Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets

kangie@monolith /data/development/temp/curl (master) $ GIT_TRACE_CURL=1 GIT_CURL_VERBOSE=1 git fetch upstream
11:07:17.509634 http.c:843              == Info: Couldn't find host github.com in the .netrc file; using defaults
11:07:17.509961 http.c:843              == Info: Host github.com:443 was resolved.
11:07:17.509967 http.c:843              == Info: IPv6: (none)
11:07:17.509970 http.c:843              == Info: IPv4: 20.248.137.48
11:07:17.509981 http.c:843              == Info:   Trying 20.248.137.48:443...
11:07:17.531516 http.c:843              == Info: Connected to github.com (20.248.137.48) port 443
11:07:17.531543 http.c:843              == Info: ALPN: curl offers h2,http/1.1
11:07:17.531674 http.c:843              == Info: rustls_connection wants us to write_tls.
11:07:17.531691 http.c:843              == Info: Curl_socket_check: reading would block
11:07:17.553442 http.c:843              == Info: rustls_connection wants us to read_tls.
11:07:17.553547 http.c:843              == Info: rustls_connection wants us to write_tls.
11:07:17.553565 http.c:843              == Info: Curl_socket_check: reading would block
11:07:17.553734 http.c:843              == Info: rustls_connection wants us to read_tls.
11:07:17.553744 http.c:843              == Info: Curl_socket_check: reading would block
11:07:17.553866 http.c:843              == Info: rustls_connection wants us to read_tls.
11:07:17.553885 http.c:843              == Info: rustls_connection_process_new_packets: invalid peer certificate: BadSignature
11:07:17.553894 http.c:843              == Info: Closing connection
fatal: unable to access 'https://github.com/curl/curl.git/': rustls_connection_process_new_packets: invalid peer certificate: BadSignature
kangie@monolith /data/development/temp/curl (master) $ curl --version                       
curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 rustls-ffi/0.12.1/rustls/0.22 zlib/1.3.1 zstd/1.5.5 c-ares/1.27.0 libpsl/0.21.5 nghttp2/1.60.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets zstd

I expected the following

A successful git https request.

curl/libcurl version

curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 rustls-ffi/0.12.1/rustls/0.22 zlib/1.3.1 zstd/1.5.5 c-ares/1.27.0 libpsl/0.21.5 nghttp2/1.60.0 Release-Date: 2024-03-27 Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets zstd

curl 8.6.0 (x86_64-gentoo-linux-musl) libcurl/8.6.0 rustls-ffi/0.10.0/rustls/0.21.0 zlib/1.3 c-ares/1.25.0 libpsl/0.21.5 Release-Date: 2024-01-31 Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp Features: alt-svc AsynchDNS HSTS HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets

operating system

Gentoo Linux

configure opts

./configure --prefix=/usr --build=x86_64-gentoo-linux-musl --host=x86_64-gentoo-linux-musl --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --datarootdir=/usr/share --disable-dependency-tracking --disable-silent-rules --disable-static --docdir=/usr/share/doc/curl-8.7.1 --htmldir=/usr/share/doc/curl-8.7.1/html --with-sysroot=/ --libdir=/usr/lib --without-ca-fallback --with-ca-bundle=/etc/ssl/certs/ca-certificates.crt --without-gnutls --without-mbedtls --without-rustls --with-rustls --with-default-ssl-backend=rustls --enable-alt-svc --enable-basic-auth --enable-bearer-auth --enable-digest-auth --enable-kerberos-auth --enable-negotiate-auth --enable-aws --enable-dict --disable-ech --enable-file --enable-ftp --disable-gopher --enable-hsts --enable-http --enable-imap --disable-ldap --disable-ldaps --enable-ntlm --disable-ntlm-wb --enable-pop3 --enable-rt --enable-rtsp --disable-smb --without-libssh2 --enable-smtp --disable-telnet --enable-tftp --enable-tls-srp --enable-ares --enable-cookies --enable-dateparse --enable-dnsshuffle --enable-doh --enable-symbol-hiding --enable-http-auth --enable-ipv6 --enable-largefile --enable-manual --enable-mime --enable-netrc --enable-progress-meter --enable-proxy --enable-socketpair --disable-sspi --disable-static --enable-pthreads --enable-threaded-resolver --disable-versioned-symbols --without-amissl --without-bearssl --without-brotli --with-fish-functions-dir=/usr/share/fish/vendor_completions.d --with-nghttp2 --without-hyper --without-libidn2 --without-gssapi --without-libgsasl --with-libpsl --without-msh3 --without-nghttp3 --without-ngtcp2 --without-quiche --without-librtmp --without-schannel --without-secure-transport --without-test-caddy --without-test-httpd --without-test-nghttpx --disable-websockets --without-winidn --without-wolfssl --with-zlib --without-zstd --with-zsh-functions-dir=/usr/share/zsh/site-functions

curl 8.6.0 (x86_64-pc-linux-gnu) libcurl/8.6.0 rustls-ffi/0.10.0/rustls/0.21.0 zlib/1.3.1 zstd/1.5.5 c-ares/1.27.0 libpsl/0.21.5 nghttp2/1.60.0
Release-Date: 2024-01-31
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets zstd
curl => /usr/bin/curl (interpreter => /lib64/ld-linux-x86-64.so.2)
    libcurl.so.4 => /usr/lib64/libcurl.so.4
        libcares.so.2 => /usr/lib64/libcares.so.2
        libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14
        libpsl.so.5 => /usr/lib64/libpsl.so.5
            libidn2.so.0 => /usr/lib64/libidn2.so.0
            libunistring.so.5 => /usr/lib64/libunistring.so.5
        librustls.so.0.10 => /usr/lib64/librustls.so.0.10
            libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/13/libgcc_s.so.1
            ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
        libm.so.6 => /usr/lib64/libm.so.6
        libzstd.so.1 => /usr/lib64/libzstd.so.1
    libz.so.1 => /usr/lib64/libz.so.1
    libc.so.6 => /usr/lib64/libc.so.6

curl 8.7.1 (x86_64-pc-linux-gnu) libcurl/8.7.1 rustls-ffi/0.12.1/rustls/0.22 zlib/1.3.1 zstd/1.5.5 c-ares/1.27.0 libpsl/0.21.5 nghttp2/1.60.0
Release-Date: 2024-03-27
Protocols: dict file ftp ftps http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smtp smtps tftp
Features: alt-svc AsynchDNS HSTS HTTP2 HTTPS-proxy IPv6 Largefile libz PSL SSL threadsafe UnixSockets zstd
curl => /usr/bin/curl (interpreter => /lib64/ld-linux-x86-64.so.2)
    libcurl.so.4 => /usr/lib64/libcurl.so.4
    libcares.so.2 => /usr/lib64/libcares.so.2
    libnghttp2.so.14 => /usr/lib64/libnghttp2.so.14
    libpsl.so.5 => /usr/lib64/libpsl.so.5
        libidn2.so.0 => /usr/lib64/libidn2.so.0
        libunistring.so.5 => /usr/lib64/libunistring.so.5
    librustls.so.0.12.1 => /usr/lib64/librustls.so.0.12.1
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/13/libgcc_s.so.1
        ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2
    libzstd.so.1 => /usr/lib64/libzstd.so.1
    libz.so.1 => /usr/lib64/libz.so.1
    libc.so.6 => /usr/lib64/libc.so.6

Are you static linking rustls with curl or using the experimental dynamic linking support?

Dynamic linking.

Does the problem reproduce with static linking? I have a hunch it won't.

This is likely unrelated to librustls and instead a regression in curl: https://github.com/curl/curl/issues/13229

It was also reported in Arch Linux for the regular curl package: https://gitlab.archlinux.org/archlinux/packaging/packages/curl/-/issues/6

Some initial investigation:

If I build tip of curl using the .so and pkg-config from rustls-ffi 0.13.0, I get a curl binary that produces the same error you observe. It happens trying to connect to https://example.com so I don't think it's related to Git endpoints but something broader:

$ LD_LIBRARY_PATH=/tmp/librustls-0.13.0/lib ./src/curl https://example.com
curl: (60) rustls_connection_process_new_packets: invalid peer certificate: BadSignature
<snipped>

If I build the tip of curl using the librustls.a from rustls-ffi 0.13.0 built with our legacy Makefile, I get a curl binary that works as expected:

$ ./src/curl https://example.com
<!doctype html>
<html>
<snipped>

If I build our client example using the .so and pkg-config from rustls 0.13.0, the client binary works as expected:

$ CA_FILE=/etc/ssl/certs/ca-certificates.crt LD_LIBRARY_PATH=/tmp/librustls-0.13.0/lib ./target/client example.com 443 /
<snipped>
HTTP/1.1 200 OK
<snipped>

I think this points towards the issue being somewhere in the curl build process :thinking:

:wave: @kpcyrd Thanks for dropping in :-D Now it's a party.

This is likely unrelated to librustls and instead a regression in curl: https://github.com/curl/curl/issues/13229

Hmm. Possible! But my testing above is also showing the error when using curl + librustls.so with a plain HTTPS server.

Would you be able to take a packet capture for this? Preferably of a TLS1.2 session -- so the certificate can be seen.

Here's a pcap from a LD_LIBRARY_PATH=/tmp/librustls-0.13.0/lib ./src/curl --tlsv1.2 https://example.com reproduction attempt from curl built with librustls 0.13.0 and dynamically linked that shows the BadSignature error:

curl.dyn.librustls.0.13.0.pcapng.tar.gz

I don't know that it's very interesting. My feeling is the issue is the build and not the host given that I can't reproduce with our own test client :thinking:

I think this points towards the issue being somewhere in the curl build process 🤔

Another data-point in this direction:

If I step back to curl @ e3a4273 when @kpcyrd updated to 0.12 I'm able to get a build that dynamically links librustls 0.13 to work without error (there were no meaningful API changes between 0.12 and 0.13 for curl).

Maybe someone can bisect what might have changed curl side that would break using rustls as a .so?

I also filed a bug about a regression in the curl build process, although it seems this was eventually fixed: https://github.com/curl/curl/issues/13200

I didn't do any testing with 0.13.0 yet.

Could you try curl 8.7.1 with https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb reverted (together with rustls-ffi 0.12.2)?

although it seems this was eventually fixed: https://github.com/curl/curl/issues/13200

My suspicion is that this fix is sufficient for the build to complete, but that it has some remaining issues when using dynamic linking with rustls.

Could you try curl 8.7.1 with https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb reverted (together with rustls-ffi 0.12.2)?

Sure!

I tried to revert https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb but there were conflicts and it's too early on a Sunday morning for me to try and figure out M4 merge conflicts :laughing:

I did the next best thing and used the parent commit before that change landed: https://github.com/curl/curl/commit/b564a5f5d5280387a88025c5f90260017847add4

Using curl at that commit + librustls 0.12.2 dynamically linked works as expected.

Repro steps:

``` # in rustls-ffi clone of 0.12.2 [daniel@blanc:~/Code/Rust/rustls-ffi]$ git rev-parse HEAD b7b316230e6a6e341e298448d8d5b1829f33bf88 [daniel@blanc:~/Code/Rust/rustls-ffi]$ cargo cinstall --release --prefix=/tmp/librustls-0.12.2 Installing pkg-config file Installing header file Installing static library Installing shared library ``` ``` # in curl clone at b564a5f [daniel@blanc:~/Code/C/curl]$ git rev-parse HEAD b564a5f5d5280387a88025c5f90260017847add4 [daniel@blanc:~/Code/C/curl]$ autoreconf -fi [daniel@blanc:~/Code/C/curl]$ LD_LIBRARY_PATH=/tmp/librustls-0.12.2/lib ./configure --with-rustls=/tmp/librustls-0.12.2 [daniel@blanc:~/Code/C/curl]$ make [daniel@blanc:~/Code/C/curl]$ LD_LIBRARY_PATH=/tmp/librustls-0.12.2/lib ./src/curl --tlsv1.2 https://example.com ... [daniel@blanc:~/Code/C/curl]$ LD_LIBRARY_PATH=/tmp/librustls-0.12.2/lib ldd ./src/.libs/curl linux-vdso.so.1 (0x00007ffff7fc8000) libcurl.so.4 => not found librustls.so.0.12.2 => /tmp/librustls-0.12.2/lib/librustls.so.0.12.2 (0x00007ffff7de3000) libpthread.so.0 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libpthread.so.0 (0x00007ffff7dde000) libdl.so.2 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libdl.so.2 (0x00007ffff7dd9000) libm.so.6 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libm.so.6 (0x00007ffff7cf7000) libz.so.1 => /nix/store/v27dxnsw0cb7f4l1i3s44knc7y9sw688-zlib-1.3/lib/libz.so.1 (0x00007ffff7cd9000) libc.so.6 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/libc.so.6 (0x00007ffff7af0000) libgcc_s.so.1 => /nix/store/a3zlvnswi1p8cg7i9w4lpnvaankc7dxx-gcc-12.3.0-lib/lib/libgcc_s.so.1 (0x00007ffff7acf000) /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib/ld-linux-x86-64.so.2 => /nix/store/1zy01hjzwvvia6h9dq5xar88v77fgh9x-glibc-2.38-44/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fca000) ```

Is this conclusive enough for us all to agree that the issue doesn't appear to be in rustls-ffi, but is something specific to curl after https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb, when using dynamic linking?

I tried to revert https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb but there were conflicts and it's too early on a Sunday morning for me to try and figure out M4 merge conflicts 😆

After some additional coffee I realized if I reverted https://github.com/curl/curl/commit/9c4209837094781d5eef69ae6bcad0e86b64bf99 and then https://github.com/curl/curl/commit/647e86a3efe1eea7a2a456c009cfe1eb55fe48eb there would be no conflicts.

Doing that with the tip of master of curl I'm able to use Rustls-ffi 0.12.2 dynamically linked without error. I'll bring this info over to https://github.com/curl/curl/issues/13200

I'm going to close this so we can consolidate discussion in https://github.com/curl/curl/issues/13200 and https://github.com/curl/curl/issues/13248

I don't believe there's any information at this point that indicates an issue with rustls-ffi itself.

Thanks all!

Circling back here to close the loop: the issue appears to have been an accidental regression in configuration upstream when using pkg-config that meant no default ca-certificate bundle was set. The BadSignature error seems confusing and certainly prolonged this investigation. I'll chase that separately over in https://github.com/rustls/rustls-ffi/issues/409.

rustls / rustls-ffi