search

solo-io / gloo

The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy
https://docs.solo.io/
Apache License 2.0
4.07k stars 435 forks source link

etag response header removed when compression is enabled #8290

Open pszeto opened 1 year ago

pszeto commented 1 year ago

Gloo Edge Version

1.14.x (latest stable)

Kubernetes Version

1.24.x

Describe the bug

Enabling compression on the gateway-proxy causes etag response headers to be removed in the client response.

Steps to reproduce the bug

  1. Enable compression on gateway. https://docs.solo.io/gloo-edge/latest/installation/advanced_configuration/gzip/
  2. Deploy an upstream that returns an etag header : https://github.com/pszeto/custom-auth-server/blob/master/deployment.yaml
  3. curl the endpoint

Expected Behavior

When curling the endpoint with compression disabled:

curl http://35.237.50.231/ -v                                                                                                                                                                                                                            
*   Trying 35.237.50.231:80...
* Connected to 35.237.50.231 (35.237.50.231) port 80 (#0)
> GET / HTTP/1.1
> Host: 35.237.50.231
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< etag: 4c52062d-f7e4-459a-9bfe-8168cb49e126
< x-auth-server: generic-server-6bb9bb54bf-z7l8j
< x-e-tag: 4c52062d-f7e4-459a-9bfe-8168cb49e126
< date: Thu, 25 May 2023 13:09:20 GMT
< x-envoy-upstream-service-time: 7
< vary: Accept-Encoding
< server: envoy
< transfer-encoding: chunked
<
* Connection #0 to host 35.237.50.231 left intact
{"message":"success"}%

etag header is present in the response. The logs shows this as well:

[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:569] [C6] message begin
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=Content-Type value=application/json
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=Etag value=4c52062d-f7e4-459a-9bfe-8168cb49e126
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=X-Auth-Server value=generic-server-6bb9bb54bf-z7l8j
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=X-E-Tag value=4c52062d-f7e4-459a-9bfe-8168cb49e126
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=Date value=Thu, 25 May 2023 13:09:20 GMT
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:820] [C6] onHeadersCompleteBase
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C6] completed header: key=Transfer-Encoding value=chunked
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:1410] [C6] status_code 200
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:1420] [C6] Client: onHeadersComplete size=6
[2023-05-25 13:09:20.203][18][trace][router] [external/envoy/source/common/router/upstream_request.cc:238] [C5][S4612556695973964163] upstream response headers:
':status', '200'
'content-type', 'application/json'
'etag', '4c52062d-f7e4-459a-9bfe-8168cb49e126'
'x-auth-server', 'generic-server-6bb9bb54bf-z7l8j'
'x-e-tag', '4c52062d-f7e4-459a-9bfe-8168cb49e126'
'date', 'Thu, 25 May 2023 13:09:20 GMT'
'transfer-encoding', 'chunked'

[2023-05-25 13:09:20.203][18][debug][router] [external/envoy/source/common/router/router.cc:1363] [C5][S4612556695973964163] upstream headers complete: end_stream=false
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=io.solo.transformation status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.grpc_web status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ext_authz status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=io.solo.filters.http.modsecurity status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=io.solo.filters.http.transformation_ee status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.cors status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.fault status=0
[2023-05-25 13:09:20.203][18][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C5][S4612556695973964163] encode headers called: filter=envoy.filters.http.compressor status=0
[2023-05-25 13:09:20.203][18][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1596] [C5][S4612556695973964163] encoding headers via codec (end_stream=false):
':status', '200'
'content-type', 'application/json'
'etag', '4c52062d-f7e4-459a-9bfe-8168cb49e126'
'x-auth-server', 'generic-server-6bb9bb54bf-z7l8j'
'x-e-tag', '4c52062d-f7e4-459a-9bfe-8168cb49e126'
'date', 'Thu, 25 May 2023 13:09:20 GMT'
'x-envoy-upstream-service-time', '7'
'vary', 'Accept-Encoding'
'server', 'envoy'

Curl endpoint with compression enabled -H "Accept-Encoding: gzip"

curl http://35.237.50.231/ -v -H "Accept-Encoding: gzip"                                                                                                                                                                                                 
*   Trying 35.237.50.231:80...
* Connected to 35.237.50.231 (35.237.50.231) port 80 (#0)
> GET / HTTP/1.1
> Host: 35.237.50.231
> User-Agent: curl/7.86.0
> Accept: */*
> Accept-Encoding: gzip
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-auth-server: generic-server-6bb9bb54bf-z7l8j
< x-e-tag: 1a523520-f205-4a2d-bc21-561f636a4436
< date: Thu, 25 May 2023 13:10:26 GMT
< x-envoy-upstream-service-time: 6
< content-encoding: gzip
< vary: Accept-Encoding
< server: envoy
< transfer-encoding: chunked
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Failed reading the chunked-encoded stream
* Closing connection 0

Logs shows the upstream is returning the etag header... but after it goes through the compressor filter. The etag is gone from the response.

[2023-05-25 13:10:26.732][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:569] [C3] message begin
[2023-05-25 13:10:26.732][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=Content-Type value=application/json
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=Etag value=1a523520-f205-4a2d-bc21-561f636a4436
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=X-Auth-Server value=generic-server-6bb9bb54bf-z7l8j
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=X-E-Tag value=1a523520-f205-4a2d-bc21-561f636a4436
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=Date value=Thu, 25 May 2023 13:10:26 GMT
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:820] [C3] onHeadersCompleteBase
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:526] [C3] completed header: key=Transfer-Encoding value=chunked
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:1410] [C3] status_code 200
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/http1/codec_impl.cc:1420] [C3] Client: onHeadersComplete size=6
[2023-05-25 13:10:26.733][19][trace][router] [external/envoy/source/common/router/upstream_request.cc:238] [C7][S5169579906438335911] upstream response headers:
':status', '200'
'content-type', 'application/json'
'etag', '1a523520-f205-4a2d-bc21-561f636a4436'
'x-auth-server', 'generic-server-6bb9bb54bf-z7l8j'
'x-e-tag', '1a523520-f205-4a2d-bc21-561f636a4436'
'date', 'Thu, 25 May 2023 13:10:26 GMT'
'transfer-encoding', 'chunked'

[2023-05-25 13:10:26.733][19][debug][router] [external/envoy/source/common/router/router.cc:1363] [C7][S5169579906438335911] upstream headers complete: end_stream=false
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=io.solo.transformation status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.grpc_web status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ext_authz status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.ratelimit status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=io.solo.filters.http.modsecurity status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=io.solo.filters.http.transformation_ee status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.cors status=0
[2023-05-25 13:10:26.733][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.fault status=0
[2023-05-25 13:10:26.736][19][trace][http] [external/envoy/source/common/http/filter_manager.cc:1129] [C7][S5169579906438335911] encode headers called: filter=envoy.filters.http.compressor status=0
[2023-05-25 13:10:26.738][19][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1596] [C7][S5169579906438335911] encoding headers via codec (end_stream=false):
':status', '200'
'content-type', 'application/json'
'x-auth-server', 'generic-server-6bb9bb54bf-z7l8j'
'x-e-tag', '1a523520-f205-4a2d-bc21-561f636a4436'
'date', 'Thu, 25 May 2023 13:10:26 GMT'
'x-envoy-upstream-service-time', '6'
'content-encoding', 'gzip'
'vary', 'Accept-Encoding'
'server', 'envoy'

Additional Context

No response

soloio-bot commented 1 year ago

Zendesk ticket #1992 has been linked to this issue.

pszeto commented 1 year ago

It looks like envoy sanitizes strong etag but preserves weak ones:

https://github.com/envoyproxy/envoy/blob/d4e8e0bff6fd76e76bf4a4073b2cdfe9fa579dcc/source/extensions/filters/http/compressor/compressor_filter.cc#L616-L629

I've updated my app to return a weak etag and confirmed that was the cause:

curl with compression enabled

curl http://35.237.50.231/ -v -H "Accept-Encoding: gzip"                                                                                                                                 
*   Trying 35.237.50.231:80...
* Connected to 35.237.50.231 (35.237.50.231) port 80 (#0)
> GET / HTTP/1.1
> Host: 35.237.50.231
> User-Agent: curl/7.86.0
> Accept: */*
> Accept-Encoding: gzip
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< etag: W/278eb1ee-99ec-4e8b-858b-d79b0f378b94
< x-auth-server: generic-server-7f9595855-9vs9p
< x-e-tag: 278eb1ee-99ec-4e8b-858b-d79b0f378b94
< date: Thu, 25 May 2023 13:44:41 GMT
< x-envoy-upstream-service-time: 6
< content-encoding: gzip
< vary: Accept-Encoding
< server: envoy
< transfer-encoding: chunked
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* Failed reading the chunked-encoded stream
* Closing connection 0

The weak etag etag: W/278eb1ee-99ec-4e8b-858b-d79b0f378b94 is present in the response.

chrisgaun commented 1 year ago

There is workaround but wondering if modify Envoy. Doesn't look if this is supported today in upstream.

chrisirhc commented 1 year ago

According to the documentation, this is expected behavior. However, I'm not sure why this option exists or is the default.

A related thread that seems to discuss this is option but for another project is at: https://bz.apache.org/bugzilla/show_bug.cgi?id=63932

github-actions[bot] commented 3 months ago

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.