Downstream connections spamming our logging servers too hard.

[akshaysgithub]

Hi,

We have Istio 1.23.2 helm-chart installed which has envoy coupled as a proxy container. And currently we are experiencing spam on our logging servers because of these 2 deprecation warnings coming from Envoy.

2024-10-15T02:40:13.507636Z warning envoy main external/envoy/source/server/server.cc:843   Usage of the deprecated runtime key overload.global_downstream_max_connections, consider switching to `envoy.resource_monitors.downstream_connections` instead.This runtime key will be removed in future.  thread=19

And

2024-10-15T02:40:31.455880Z warning envoy main external/envoy/source/server/server.cc:936   There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.    thread=12

For both of them, I am setting values in Proxy section of values.yaml, but they are not taking effect(pods have been restarted). Can someone tell me how to stop this warning for good.

Values.yaml

global:
  proxy:
    resourceMonitors:
      - name: envoy.resource_monitors.downstream_connections
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.resource_monitors.downstream_connections.v3.DownstreamConnectionsConfig
          max_active_downstream_connections: 10000

I have spent a lot of man-hours fixing this issue, but no luck.

Any idea would be really helpful on how to block this from Envoy side completely.

Thanks a lot. :-)

[tyxia]

These two are legit warnings as those feature will be deprecated. I don't why we should mute them

cc @nezdolik who added those and is the owner of resource monitor.

[akshaysgithub]

@tyxia Those might be legit warnings, but we are depending upon the bundled version of Envoy inside Istio. So, we are not actually able to change them, and a warning sometimes is okay, but it spams our logging server. More than 70k warnings in days, that's a bit too much. :-)

So, we are stuck with something we cannot modify and cannot disable the warning as well.

Any idea how we can disable it?

[ramaraochavali]

More than 70k warnings in days, that's a bit too much. :-)

Is this across all your services? AFAIK, it only comes up during start up. Curious why are you seeing so many of them

[akshaysgithub]

@ramaraochavali It's for all our services, non-stop, as it's from sidecar containers of Istio-proxy aka Envoy. :-) Not just during pod startup.

The attached screenshot is for last 1 month for those warnings for dev, more on Prod.

[Pawan-Bishnoi]

On a quick try I couldn't reproduce this. I installed istio 1.23.2 using istioctl. Will take a deeper look tomorrow.

istioctl install --set profile=demo

istioctl version
client version: 1.23.2
control plane version: 1.23.2
data plane version: 1.23.0 (5 proxies), 1.23.2 (3 proxies)

attaching the logs.txt

[akshaysgithub]

We are installing Istio via Helm-chart, ofcourse this was also during operator, which we had before. Second, does the default revision has any changes? We are using the default revision. :-)

[nezdolik]

@akshaysgithub those error messages are being produced on server startup. Please attach Envoy config from any Envoy pod that is having issue. We cannot help with troubleshooting Istio side, so need to make sure that resulting Envoy config looks correct.

[kyessenov]

CC @kyessenov