Web App Access: fails with Gzip Content-Encoding

Internal Server Error occurs when a user tries to connect to a gzip-encoded endpoint via Web App Access. Browser, cURL, and Go (net/http with TLSClient) successfully connect and decode the body from the same endpoint.

Expected behavior: Web App Access successfully proxies connections to endpoints with gzip content encoding.

Current behavior: Web App Access returns an Internal Server Error for such endpoints.

Steps to reproduce

Obtain an example endpoint with Content-Encoding: gzip. You can use example server attached below.
Add a configuration entry for Web App Access pointing to this server.
Open this resource via Teleport in your browser.

Details

Version

# teleport version
Teleport v14.1.1 git:api/v14.1.1-0-gfb6429e go1.21.3

# uname -a
Linux example-node 5.4.0-165-generic #182-Ubuntu SMP Mon Oct 2 19:43:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:    20.04
Codename:   focal

Error log

2023-10-26T21:31:43+03:00 ERRO [APP:SERVI] Error forwarding to https://127.0.0.1:3443/, err: dial tcp 127.0.0.1:3443: connect: connection refused reverseproxy/reverse_proxy.go:231

Configuration entry

app_service:
  ...
  apps:
    - name: test-gzip
      uri: "https://127.0.0.1:3443"
      insecure_skip_verify: true

Example endpoint server

You might need an example server to reproduce this issue. Golang is outside of my knowledge, I can only provide a zero-deps Node.js example.

Requirements: Node.js 16 or newer.

const zlib = require('zlib');
const https = require('https');

const key = `-----BEGIN PRIVATE KEY-----
MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQCRD/mtxsEUidiD
A4Uk23Emul3sSP9tTvkJ8iuiMT4XcgnzTeqJWx15v0hZF1+e5oQKevT2hA+kIABm
0lQEg8M7dDPPvdhTz0ew6ezDeR3itbs/YpDaHm9zCQmeEjWIEmD9GWk36vsm8s+0
2L4WOsQ1x98TFaYopcYHL3W+79YTbzEYX+y+fGNbz/gLyGKlYj0WqWQaz+uJcE0P
r+6VdHso4e+YKQxjNJGhOfofdhZ+dZPkjpJZB/PxGixAnh/T4enz962mSJ7zeDWL
2tlWt0GQhod5G86zzUAQbb1n31ObhUu4r6Dd2QxshAYTfs9aHzSjpHczPVe+o8w7
cYI7hTeoCLmvAtLNG3HxKuSqzIoS+e5kf5tkVGaZu2HhDrvfq4CeqiUi5iDIxhpr
n5LwMIL6eMbi2m9r0KYae3Wy7VuY+CgT9HPoG2H5CQTmzB/Q3eyv/LPxVeQ/QpBm
56BUBTrKp0rh5WiPbYM3UP8buOLKATeRVDwOXH7hUceLNVLVgsswYBQ1iDE6xQqR
fYuB7d8WHioUke/A4WnFZKZufXYS5BcgeIyYtrdJ6UNRQCQHjgi6MyNIXMG1b9I6
hhC+KSL3YDV2SBH8Oi4uPqBH89Fni+QKeT4r/aKnaG1Hu0copsRk90oZ5/PrkydW
tv2FFBKLUsk5F6SAKrYQ1YUM5K3HzQIDAQABAoICAATjkbHkYrH/QlOKDRT0q1/X
hJ5XyHoF4RlPdt6Aw9u07XZsgBTQhBIMGxNbmQ6pLLaQtcxQH5nTtUfB23CKg0wv
6qYb+xw5/mvNOnxNCu+O2eh1OTYMtAtfuNoBuZmkKu4kytnGWuARa2QoMOHdaar5
WUCSqHayvZWU6SY9/YxKoVA1vWwfE5cuz2iQQJjb0ny+yT+pFcJcTVCTyJnzYso6
MSHC8mauiSDyrAYe6/unmUwo7PBAtx0/WVH1MlPpB4JALPNPe8HvYXic79m/E9/b
1NmFUSZYqsnS9QD48x+o8qgfDTAk5kkK+wxxp+nc5uCEBLtunBQx/4paVT4Ny5Ya
cgJjd5wokQgTbZLRBPHC2H6qiLOpCDd3Oa2bEjFfpM1q5sBcUdXPq+v5I5gWwjUF
U93cG0xEsYS0qiBfqUxSNS/kEwwJ28d/9QhyM5a1cuG2z2vTqnH4WNoQ5fLyUf4f
h4zKDm9KH1MocYsX6v1aNHvpW48WEPFPWXYI1eQZxdBm8dx9wb/QAwfm4N3KJru7
J9NLlPZPLBCTJ8D5L0+RbP3mbvAs6rm+ng8liRhrg4Zmr6BAsUWeqz0YADb+CIHj
eeUfWkNdj6/8BBzRH/LJSAg7nhFqcrJ58bTDosbTlIHjqK3HykEYXILNAnhkXKB1
VXqgvbYXhCW3mcKYZ/YRAoIBAQDKMvCb+vmaA82vTYs595nozsVw8m675Nx/Iiom
1Uj8053aTBPNLumDwPE57A6lgUiWKo+dP/k7uHKh52+8wPrM8OJr2NsF2djdx+/G
bm2S2Fuh5v56dKeSLVGBfv8PJ6AdMpmHcFuacRsF6F9XyOhmCNjKSRUN190XHcbF
sxdHnckNTB1QmvubDjf8S4BEkILZsuVAJeBBb6mGV9ZeqqfppVvwLgBEH9HjMXxh
wR2bu80+9yBTkUS8w8ilRod+/0AU14UlAK5QEx4fCWdZnxijR8F4BvaKH19kHpcP
6j1mCf+hsQFwrQsejWMLf6dKsfI5Ic0DkdLQhsVNyszKoTK9AoIBAQC3qRljVlHC
1dN06JqBatSngaxw0N2E0U5/QL+Ka2th9PFQtklqzq4z3RkII71elONI4PnLUllJ
eLFNvRUsEFkXE5mVemHoJeK2jKFXzlDjrwpqX7HL9dmaj15Up/iG0LtFMxsugqqA
kFMzN/AMIoN/6AkjdzJCRsYSTDB7G2PACmjjFDEuMyu/wAhEg9Gs9IhT2FiQBwdZ
jT9lYef0krZRC7Tk/pYleE0hAsM2J5OsXA+OJWIYXp7HEOgDWeAsJ0KVE8C4C3R4
8xmpalUHLBa6J5HNohEzZ9I24LQMg31sNfsISHaciRl3V33pEACE2QNFh8kBv//+
VUi4Q7AweEJRAoIBACQOt3epZu9/NCPJ2UUqEWlBzlXY9cROCMAnMIsiR9TGTAUM
wg86CWx0cNAad3YuOuqEhn3NVb2/o7mZkwR/CXqYjlgbINvHQHQ0Xl7tfF7OdHDH
x6PAv81ChaFpK0ThlRz/0/8iG0PMXbxM9wupiUAA3eAc7kM8tnGT3smMqV8dFSho
5WVlbe71tygnsqUHfD4kN9ubpYUly35olZLmLfFtipqnKWb8oao5PewUyxnlgLF0
0m/0nORoqZRHoowUKrJMyLKTj4I6FooEXEjQJU2g0zB7VWXGOHWbOuccp3xV1h9Y
eGBGi5AUOJBpYETivMZRb8TuMUD8d5zqyGBuJKUCggEASTIciymQxky/zJ0TTrbL
78JPflrPnipo1XBxpfHvxwd+0UME+YtLhCH+4u6Tf0lpV9dJ1o2IRi7Swv+zm28c
goTuNdOjh4jp1QUxHDR+qm9NgF2JF5dw1ReEacrT0q8Ho6mdkKjE2R4r+oK1qbBr
piNM2RSdNCrOr+YrgDTHSXccnDuk6hu8uNwIr321p++BEaFskmIqvLb9aCNf9Wov
PRekTYtlPaO3YYufT0rwoMXD8UvWz42OTBnhzcwY+p/e+lj3+qGeFsIUHR/iqUwZ
9rDkfAa2Qj2DWxmsyDvzVZgh5wgyNWqyeoylAqi8/DUdB635HzhtBujoZX+cjYSw
MQKCAQEAoTsZtz1c63n/Lvz2BG2Ffr8BIuCW+UtD62AoiOygJJ5ROSBLVwR/lIiD
gZlaTUGvzO7zE+VUdjhU9bc0mQQVuabonBFxo+ggMGyXNZQMB1UyuOkJlcRfQNMX
6RE/hjJz18GQGAObgQqZDxzzLXps7ThUqpSqFlI9r2m8lck1qSkSdGWA8j+g1ZVH
7LL9uY15GyIlat6IOcerM/lK7LsThqnjQn5C0Gqrh9ma6uIWJMDdCGzMQp5DViC6
sFJofYTi6+VlgWtgqwl8DXHsbAU9d5rYUuKem6O8eXydKigYvtGXVf+As8h43cIB
1MulIuQWWj0g9rpttlvJoZfPk7b4TQ==
-----END PRIVATE KEY-----
`;

const cert = `-----BEGIN CERTIFICATE-----
MIIFpzCCA4+gAwIBAgIUHNgXXh3mEOqRJpncOeT03VoUYHYwDQYJKoZIhvcNAQEL
BQAwYzELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEcMBoGA1UEAwwTZ3ppcC10ZXN0Lmxv
Y2FsaG9zdDAeFw0yMzEwMjYxODI4MTZaFw0yNDEwMjUxODI4MTZaMGMxCzAJBgNV
BAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBX
aWRnaXRzIFB0eSBMdGQxHDAaBgNVBAMME2d6aXAtdGVzdC5sb2NhbGhvc3QwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCRD/mtxsEUidiDA4Uk23Emul3s
SP9tTvkJ8iuiMT4XcgnzTeqJWx15v0hZF1+e5oQKevT2hA+kIABm0lQEg8M7dDPP
vdhTz0ew6ezDeR3itbs/YpDaHm9zCQmeEjWIEmD9GWk36vsm8s+02L4WOsQ1x98T
FaYopcYHL3W+79YTbzEYX+y+fGNbz/gLyGKlYj0WqWQaz+uJcE0Pr+6VdHso4e+Y
KQxjNJGhOfofdhZ+dZPkjpJZB/PxGixAnh/T4enz962mSJ7zeDWL2tlWt0GQhod5
G86zzUAQbb1n31ObhUu4r6Dd2QxshAYTfs9aHzSjpHczPVe+o8w7cYI7hTeoCLmv
AtLNG3HxKuSqzIoS+e5kf5tkVGaZu2HhDrvfq4CeqiUi5iDIxhprn5LwMIL6eMbi
2m9r0KYae3Wy7VuY+CgT9HPoG2H5CQTmzB/Q3eyv/LPxVeQ/QpBm56BUBTrKp0rh
5WiPbYM3UP8buOLKATeRVDwOXH7hUceLNVLVgsswYBQ1iDE6xQqRfYuB7d8WHioU
ke/A4WnFZKZufXYS5BcgeIyYtrdJ6UNRQCQHjgi6MyNIXMG1b9I6hhC+KSL3YDV2
SBH8Oi4uPqBH89Fni+QKeT4r/aKnaG1Hu0copsRk90oZ5/PrkydWtv2FFBKLUsk5
F6SAKrYQ1YUM5K3HzQIDAQABo1MwUTAdBgNVHQ4EFgQU4jNT8/j9fe4y0Tiy8i8z
H8vO9sswHwYDVR0jBBgwFoAU4jNT8/j9fe4y0Tiy8i8zH8vO9sswDwYDVR0TAQH/
BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAgEAf1CXhmSRakFacWaHzMcr/xvRJses
YABjxdMSxIQvot1tAw5HATje20GiUks88WjTf786en+Cepgx4+/mYC5/h9pTNKtT
K/yggeeq508Z9ce60e2dl1+DekVZO7yicadOU6lxTFgUJFNx/EuEyqM3ilxdk6rv
krzAEngsGc3liG7/SvEvtGn2DcPmEhyckNn61cPLHRFBga9lHQ5yqB7AONs/f3/d
UgLB4JgsM4zDXuRCahw/l1ZNlJwyNGM1eAH/0yg0WitWHI6BvleAJj7ZPiDA3Idd
nbYtSLh0HMxuwz+ZYqbBpdpKls/zg4jHtmBHCxXQ9FN6Ont0czMjasNRNbPsTI80
OMgDt4JXVj/Zxt5l7YQLszbZyFtpHoMXhk+y0AH6LeL9SLTgEp0qGk9PPWzIcVRV
4PLGv1lWj6SQe9Rc20QWDxvA0eG0gtunCyjeISbuNuCjW1DQkjyrBP6v0Gnm0W4h
CGqPYuUqyLpKPgCJhXOptb+Zw066Zb11QnhRedbx5Hh0qDQGtvuwrMUxF/VXkZc/
S/dBnVApJ5F+5dD2B0mel3SaDVIF2RoCMmkmzxQDX56MqEgfZN6PsvNJkRggqSHZ
VrG4WEihMCp/DSgP63KilqhDauKvv6WLaiib2fhKY0IO59Nuo2KwBMTNAliiIS4W
saA0IsdaGexa1+c=
-----END CERTIFICATE-----
`;

https.createServer({ key, cert }, (req, res) => {
  res.writeHead(200, { 'content-encoding': 'gzip' });
  zlib.createGzip().end('<h1>Hello, Secure World!</h1>').pipe(res)
}).listen(3443);

Example cURL request

curl -kv --compressed https://localhost:3443
*   Trying 127.0.0.1:3443...
* Connected to localhost (127.0.0.1) port 3443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=gzip-test.localhost
*  start date: Oct 26 18:28:16 2023 GMT
*  expire date: Oct 25 18:28:16 2024 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=gzip-test.localhost
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* using HTTP/1.1
> GET / HTTP/1.1
> Host: localhost:3443
> User-Agent: curl/8.1.2
> Accept: */*
> Accept-Encoding: deflate, gzip
> 
< HTTP/1.1 200 OK
< content-encoding: gzip
< Date: Thu, 26 Oct 2023 19:09:25 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
< 
* Connection #0 to host localhost left intact
<h1>Hello, Secure World!</h1>

Example Go client

I've used it to verify that the issue is not in crypto/tls or net/http modules.

package main
import (
    "crypto/tls"
    "fmt"
    "io"
    "net/http"
)
func httpsClient(url string) []byte {
    tr := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    }
    client := &http.Client{Transport: tr}
    resp, err := client.Get(url)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    fmt.Println("Status =", resp.Status)
    msg, _ := io.ReadAll(resp.Body)
    return msg
}

func main() {
    msg := httpsClient("https://localhost:3443")
    fmt.Println("Body =", string(msg))
}

gravitational / teleport