search

Mbed-TLS / mbedtls

An open source, portable, easy to use, readable and flexible TLS library, and reference implementation of the PSA Cryptography API. Releases are on a varying cadence, typically around 3 - 6 months between releases.
https://www.trustedfirmware.org/projects/mbed-tls/
Other
5.5k stars 2.6k forks source link

ssl_client2 returns "HTTP/1.1 421 Misdirected Request" when making a GET request to https://ftp.debian.org #9093

Closed jetm closed 6 months ago

jetm commented 6 months ago

Summary

I am working in a HTTPS client using LWIP with mbedTLS, and when trying to access some HTTPS servers, like https://ftp.debian.org it returns HTTP/1.1 421 Misdirected Request. It's reproducible with ssl_client2 too. With curl, it produces the expected output.

I am not sure what I am missing, if it's a mbedTLS configuration/implementation or an HTTP client implementation.

ssl_client2 output

$ ssl_client2 server_name=ftp.debian.org server_port=443 ca_file=/etc/ssl/certs/ca-certificates.crt
build version: Mbed TLS 3.6.0 (build 50724864)

  . Seeding the random number generator... ok
  . Loading the CA root certificate ... ok (0 skipped)
  . Loading the client cert. and key... ok (key type: RSA)
  . Setting up the SSL/TLS structure...
Run mbedtls_ssl_set_hostname OK
 ok
  . Connecting to tcp/ftp.debian.org/443... ok
  . Performing the SSL/TLS handshake...
Verify requested for (Depth 2):

Verify requested for (Depth 1):

Verify requested for (Depth 0):
 ok
    [ Protocol is TLSv1.2 ]
    [ Ciphersuite is TLS-ECDHE-RSA-WITH-CHACHA20-POLY1305-SHA256 ]
    [ Key size is 256 ]
    [ Record expansion is 21 ]
    [ Maximum incoming record payload length is 16384 ]
    [ Maximum outgoing record payload length is 16384 ]
  . Verifying peer X.509 certificate... ok
  . Peer certificate information    ...
cert. version     : 3
serial number     : 04:02:4A:F4:58:69:AF:5D:8F:2D:10:47:DE:68:71:B2:D6:14
issuer name       : C=US, O=Let's Encrypt, CN=R3
subject name      : CN=cdn-fastly.deb.debian.org
issued  on        : 2024-03-12 00:46:35
expires on        : 2024-06-10 00:46:34
signed using      : RSA with SHA-256
RSA key size      : 2048 bits
basic constraints : CA=false
subject alt name  :
    dNSName : cdn-fastly-v6.deb.debian.org
    dNSName : cdn-fastly.deb.debian.org
    dNSName : deb.debian.org
    dNSName : ftp.debian.org
    dNSName : ftp.ipv6.debian.org
    dNSName : httpredir.debian.org
key usage         : Digital Signature, Key Encipherment
ext key usage     : TLS Web Server Authentication, TLS Web Client Authentication
certificate policies : ???

  > Write to server: 34 bytes written in 1 fragments

GET / HTTP/1.1
Extra-header:

  < Read from server: 158 bytes read

HTTP/1.1 421 Misdirected Request
Connection: keep-alive
Content-Length: 297
content-type: text/plain; charset=utf-8
x-served-by: cache-pdk-kpdk1780136

  . Closing the connection... done

curl output

$ curl --tls-max 1.2 -v https://ftp.debian.org
* Host ftp.debian.org:443 was resolved.
* IPv6: 2a04:4e42:77::644
* IPv4: 151.101.54.132
*   Trying 151.101.54.132:443...
* Connected to ftp.debian.org (151.101.54.132) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=cdn-fastly.deb.debian.org
*  start date: Mar 12 00:46:35 2024 GMT
*  expire date: Jun 10 00:46:34 2024 GMT
*  subjectAltName: host "ftp.debian.org" matched cert's "ftp.debian.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://ftp.debian.org/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: ftp.debian.org]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: ftp.debian.org
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< server: Apache
< x-content-type-options: nosniff
< x-frame-options: sameorigin
< referrer-policy: no-referrer
< x-xss-protection: 1
< permissions-policy: interest-cohort=()
< last-modified: Fri, 01 Sep 2023 16:12:16 GMT
< etag: "754-6044e6c99e42d"
< x-clacks-overhead: GNU Terry Pratchett
< content-type: text/html
< accept-ranges: bytes
< date: Thu, 02 May 2024 23:34:34 GMT
< via: 1.1 varnish
< age: 0
< x-served-by: cache-pdk-kpdk1780080-PDK
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1714692874.077409,VS0,VE207
< vary: Accept-Encoding
< content-length: 1876
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
   <TITLE>Debian mirrors backed by Fastly CDN</TITLE>
</HEAD>
<BODY>

<H1>Debian mirrors backed by Fastly CDN</H1>

<p>
This is <code>deb.debian.org</code>.

This service provides mirrors for the following Debian archive repositories:
<ul>
  <li><code>/debian/</code></li>
  <li><code>/debian-debug/</code></li>
  <li><code>/debian-ports/</code></li>
  <li><code>/debian-security/</code></li>
  <li><code>/debian-security-debug/</code></li>
</ul>

<p>
The server <code>deb.debian.org</code> does not have packages itself, but the
name has SRV records in DNS that lets apt <em>in stretch and later</em> find places.

<p>
To use it with a sufficiently recent apt, you for instance can put
<pre>
deb http://deb.debian.org/debian unstable main
deb http://deb.debian.org/debian-debug unstable-debug main
deb http://deb.debian.org/debian-ports unstable main
</pre>
in your <code>sources.list</code>.

<p>
As of July 2022 the SRV record is
<pre>
_http._tcp.deb.debian.org.    IN      SRV     10 1 80 prod.debian.map.fastly.net.
</pre>

If you hit the server behind <code>deb.debian.org</code> directly, either because you
use an older apt or because you use a HTTP proxy that does not support SRV records,
your requests will get HTTP redirected to one of the CDN instances.

If you want to avoid the redirects, you can pick one instance directly.  For instance,
this also works in your <code>sources.list</code>:
<pre>
deb http://cdn-fastly.deb.debian.org/debian stable main
deb http://cdn-fastly.deb.debian.org/debian-security stable-security main
deb http://cdn-fastly.deb.debian.org/debian-security-debug stable-security-debug main
</pre>

This service is sponsored by <a href="https://www.fastly.com/">[Fastly](https://www.fastly.com/)</a>.

<P>
<HR NOSHADE />
<FONT size="-1"><a href="mailto:dsa@debian.org">[DSA](mailto:dsa@debian.org)</a></FONT>

</BODY>
</HTML>
* Connection #0 to host ftp.debian.org left intact

System information

Mbed TLS version (number or commit id): Mbed TLS 3.6.0 (build 50724864) Operating system and version: OS: Arch Linux Linux x86_64, Kernel: 6.8.8-2-cachyos Configuration (if not default, please attach mbedtls_config.h): mbedtls_config.h.zip Compiler and options (if you used a pre-built binary, please indicate how you obtained it): N/A Additional environment information:

Expected behavior

curl --tls-max 1.2 -v https://ftp.debian.org or a browser output.

Actual behavior

Explained in the summary

Steps to reproduce

cmake -S . -B build -G Ninja && cmake --build build --clean-first && ./build/programs/ssl/ssl_client2 server_name=ftp.debian.org server_port=443 ca_file=/etc/ssl/certs/ca-certificates.crt

Additional information

jetm commented 6 months ago

OK, I think found the problem. It was missing Host: in the GET request. With the following change, I can get an HTTP/1.1 200 OK.

$ git --no-pager diff programs/ssl/ssl_client2.c
diff --git a/programs/ssl/ssl_client2.c b/programs/ssl/ssl_client2.c
index 43133d901..97dfde663 100644
--- a/programs/ssl/ssl_client2.c
+++ b/programs/ssl/ssl_client2.c
@@ -108,7 +108,7 @@ int main(void)
 #define DFL_SRTP_MKI            ""
 #define DFL_KEY_OPAQUE_ALG      "none"

-#define GET_REQUEST "GET %s HTTP/1.0\r\nExtra-header: "
+#define GET_REQUEST "GET %s HTTP/1.0\r\nHost: %s\r\nExtra-header: "
 #define GET_REQUEST_END "\r\n\r\n"

 #if defined(MBEDTLS_SSL_HANDSHAKE_WITH_CERT_ENABLED)
@@ -724,7 +724,7 @@ static int build_http_request(unsigned char *buf, size_t buf_size, size_t *reque
     int ret = MBEDTLS_ERR_ERROR_CORRUPTION_DETECTED;
     size_t len, tail_len, request_size;

-    ret = mbedtls_snprintf((char *) buf, buf_size, GET_REQUEST, opt.request_page);
+    ret = mbedtls_snprintf((char *) buf, buf_size, GET_REQUEST, opt.request_page, opt.server_name);
     if (ret < 0) {
         return ret;
     }