Response is not received by the clients

[Salah-Rida]

Title: Response is not received by the client

Description:

Hello,

We are experiencing an issue with Envoy Proxy, which is likely related to the default configuration.

The problem is that clients do not receive a response until Envoy reaches the request timeout, which is currently set to 1 hour. The request itself takes approximately 15 minutes to process (as seen in our service logs where the request is completed and results are returned), so duration doesn’t seem to be the issue. I also tested the same request by bypassing Envoy Proxy, and the response body was received correctly.

It may be worth mentioning that the response byte size is 3,479,806. When we run smaller requests, we are able to receive the response through Envoy Proxy without issues.

can you please point me the config I need to change, I didn't find something like response body size limit in your doc ?

salah

[KBaichoo]

What version of envoy are you using? What version of HTTP is your client using? What is your envoy config?

You could gather access log: https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage or enable more verborse logging https://www.envoyproxy.io/docs/envoy/latest/operations/cli.html#cmdoption-l

[KBaichoo]

cc @tyxia

[Salah-Rida]

Hello, ​ envoy version : 1.30_distroless ​ ​ envoy config : ​ cds.yaml ############################################ ` resources:

• "@type": type.googleapis.com/envoy.config.cluster.v3.Cluster name: gcp_metadata_server connect_timeout: 5000s type: strict_dns lb_policy: round_robin load_assignment: cluster_name: gcp_metadata_server endpoints:
  • lb_endpoints:
    • endpoint: address: socket_address: address: metadata.google.internal port_value: 80 ​
• '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster circuit_breakers: thresholds:
  • max_connections: 100000 max_pending_requests: 100000 max_requests: 100000 max_retries: 3
  • max_connections: 100000 max_pending_requests: 100000 max_requests: 100000 priority: HIGH ​ typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions" common_http_protocol_options: idle_timeout: 0s explicit_http_config: http_protocol_options: {} connect_timeout: 120s dns_lookup_family: V4_ONLY type: LOGICAL_DNS dns_refresh_rate: 100s load_assignment: cluster_name: service_python-fastapi-dev-0-cloudrun endpoints:
  • lb_endpoints:
    • endpoint: address: socket_address: address: python-fastapi-dev-0-xxxxxxxxxxxxx.a.run.app port_value: 443 metadata: typed_filter_metadata: envoy.filters.http.gcp_authn: '@type': type.googleapis.com/envoy.extensions.filters.http.gcp_authn.v3.Audience url: https://python-fastapi-dev-0-xxxxxxxxxxxxx.a.run.app name: service_python-fastapi-dev-0-cloudrun transport_socket: name: envoy.transport_sockets.tls typed_config: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext common_tls_context: alpn_protocols:
    • HTTP/2 validation_context: trusted_ca: filename: /etc/ssl/certs/ca-certificates.crt sni: python-fastapi-dev-0-xxxxxxxxxxxxx.a.run.app ​
• '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster circuit_breakers: thresholds:
  • max_connections: 100000 max_pending_requests: 100000 max_requests: 100000 max_retries: 3
  • max_connections: 100000 max_pending_requests: 100000 max_requests: 100000 priority: HIGH connect_timeout: 120s dns_lookup_family: V4_ONLY dns_refresh_rate: 100s http2_protocol_options: {} load_assignment: cluster_name: service_python-fastapi-dev-1-gke endpoints:
  • lb_endpoints:
    • endpoint: address: socket_address: address: python-fastapi-dev-1.gke_cluster.gcp_project_id.dev.gcp.cloud.uk port_value: 443 name: service_python-fastapi-dev-1-gke transport_socket: name: envoy.transport_sockets.tls typed_config: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext common_tls_context: alpn_protocols:
    • h2 sni: python-fastapi-dev-1.gke_cluster.gcp_project_id.dev.gcp.cloud.uk type: STRICT_DNS ​ **lds.yaml #############################################** resources:
• '@type': type.googleapis.com/envoy.config.listener.v3.Listener name: listener_1 address: socket_address: address: 0.0.0.0 port_value: 7070 filter_chains:
  • filters:
    • name: envoy.filters.network.http_connection_manager typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager access_log:
      • filter: not_health_check_filter: {} name: envoy.access_loggers.billing typed_config: '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog log_format: json_format: email: '%REQ(X-User-Email)%' endpoint: '%REQ(x-envoy-original-path)%' log_name: billing request_duration: '%DURATION%' request_method: '%REQ(:METHOD)%' response_code: '%RESPONSE_CODE%' service-name: '%UPSTREAM_CLUSTER%' user_id: '%REQ(X-User-Id)%' path: /dev/stdout stat_prefix: ingress_http stream_idle_timeout: 3600s route_config: name: local_route virtual_hosts:
        • domains:
        • '*' name: local_service routes:
        • match: prefix: /python-fastapi-dev-0/ route: cluster: service_python-fastapi-dev-0-cloudrun host_rewrite_literal: python-fastapi-dev-0-xxxxxxxxxxxxx.a.run.app prefix_rewrite: / timeout: 0s
        • match: prefix: /python-fastapi-dev-1/ route: cluster: service_python-fastapi-dev-1-gke host_rewrite_literal: python-fastapi-dev-1.gke_cluster.gcp_project_id.dev.gcp.cloud.uk prefix_rewrite: / timeout: 0s ​ http_filters:
      • name: envoy.filters.http.jwt_authn typed_config: '@type': type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication providers: keycloak: forward: true forward_payload_header: x-jwt-payload from_headers:
        • name: Authorization value_prefix: 'Bearer ' issuer: https://keycloak:8443/realms/project payload_in_metadata: verified_jwt remote_jwks: cache_duration: 300s http_uri: cluster: keycloak timeout: 5s uri: https://keycloak:8443/realms/project/protocol/openid-connect/certs rules:
          • match: safe_regex: regex: ^/app-uat-2/api.*$ requires: provider_name: keycloak
      • name: envoy.filters.http.lua typed_config: '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua inline_code: "function envoy_on_request(request_handle)\n\tlocal meta\ \ = request_handle:streamInfo():dynamicMetadata()\n\theaders = request_handle:headers()\n\ \tfor key, value in pairs(meta) do\n\t headers:add(\"X-User-Id\"\ , value.verified_jwt.preferred_username)\n\t headers:add(\"X-User-Email\"\ , value.verified_jwt.email)\n headers:add(\"X-User-Groups\"\ , table.concat(value.verified_jwt.groups, \",\"))\n headers:add(\"\ X-User-AuthURL\", \"https://keycloak:8443/realms/project\"\ \ )\n\tend\n local path = headers:get(\":path\")\n if path\ \ ~= nil and path ~= '' then\n headers:add(\"X-Forwarded-Prefix\"\ , string.match(path, \"(/[^/]+)\") )\n end\nend\n"
      • name: envoy.extensions.filters.http.gcp_authn typed_config: '@type': type.googleapis.com/envoy.extensions.filters.http.gcp_authn.v3.GcpAuthnFilterConfig http_uri: cluster: gcp_metadata_server timeout: 100s uri: http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=[AUDIENCE] token_header: name: X-Serverless-Authorization value_prefix: 'Bearer '
      • name: envoy.filters.http.router typed_config: '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router ​ transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificates:
        • certificate_chain: filename: "/etc/envoy/envoy.pem" private_key: filename: "/etc/envoy/envoy-priv.pem" validation_context: trusted_ca: filename: /etc/envoy/CA.pem **server config** node: cluster: test-cluster id: test-id dynamic_resources: lds_config: path: "/etc/envoy/lds.yaml" cds_config: path: "/etc/envoy/cds.yaml" admin: access_log_path: /tmp/admin_access.log address: socket_address: protocol: TCP address: 0.0.0.0 port_value: 7000 ` ​ ​ ​ logs : nothing special, no errors just waiting for response. ​ [2024-11-13 11:41:37.722][1][trace][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:104] starting async DNS resolution for metadata.google.internal [2024-11-13 11:41:37.722][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:369] dns resolution for metadata.google.internal started [2024-11-13 11:41:37.723][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:37.723][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:37.792][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:37.796][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:37.799][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:159] dns resolution without records for metadata.google.internal [2024-11-13 11:41:37.799][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:37.799][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:289] dns resolution for metadata.google.internal completed with status 0 [2024-11-13 11:41:37.800][1][trace][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:112] async DNS resolution complete for metadata.google.internal [2024-11-13 11:41:37.800][1][debug][upstream] [source/common/upstream/upstream_impl.cc:458] transport socket match, socket default selected for host with address 169.254.169.254:80 [2024-11-13 11:41:37.800][1][debug][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:177] DNS refresh rate reset for metadata.google.internal, refresh rate 5000 ms [2024-11-13 11:41:40.270][1][debug][main] [source/server/server.cc:241] flushing stats [2024-11-13 11:41:42.801][1][trace][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:104] starting async DNS resolution for metadata.google.internal [2024-11-13 11:41:42.801][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:369] dns resolution for metadata.google.internal started [2024-11-13 11:41:42.801][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:42.802][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:42.805][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:42.809][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:42.811][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:159] dns resolution without records for metadata.google.internal [2024-11-13 11:41:42.811][1][trace][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:332] Setting DNS resolution timer for 5000 milliseconds [2024-11-13 11:41:42.812][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:289] dns resolution for metadata.google.internal completed with status 0 [2024-11-13 11:41:42.812][1][trace][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:112] async DNS resolution complete for metadata.google.internal [2024-11-13 11:41:42.812][1][debug][upstream] [source/common/upstream/upstream_impl.cc:458] transport socket match, socket default selected for host with address 169.254.169.254:80 [2024-11-13 11:41:42.812][1][debug][upstream] [source/extensions/clusters/strict_dns/strict_dns_cluster.cc:177] DNS refresh rate reset for metadata.google.internal, refresh rate 5000 ms [2024-11-13 11:41:45.275][1][debug][main] [source/server/server.cc:241] flushing stats ​ ​ ​ the behavior is the next : we send a sync request that takes more than 10 min, in application log we see that the response is returned, but the downstream do not receive the response. when sync request take less than 10 min, downstream receive a 200 response with response body. When we bypass envoy and send request a sync request that takes more than 10 min directly to service url, response is received ( envoy issue ). ​ ​ ​ ​ ​ ​ I have run couple of additional tests, therefore it seems that we face an around 10 min timeout. our target is to allow our users to be able to run max 1h sync request. ​ ​ for your attention : @tyxia @KBaichoo