Issues while migrating to proxy-protocol pod annotation

[himanshujaindev]

Is this the right place to submit this?

• [X] This is not a security vulnerability or a crashing bug
• [X] This is not a question about how to use Istio

Bug Description

To avoid downtime, we need to first move to pod annotation and then behind the scenes delete the envoyfilter.

When both are used at the same time, the below error is seen in ingressgateway pod (when debug logs are enabled)

{"level":"debug","time":"2024-03-12T11:49:14.458353Z","scope":"envoy filter","msg":"failed to read proxy protocol (exceed max v1 header len)","caller":"external/envoy/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc:519","thread":22}

Reference: https://istio.io/latest/docs/ops/common-problems/upgrade-issues/#use-gateway-topology-to-enable-proxy-protocol-on-the-ingress-gateways

Version

$ istioctl version
client version: 1.20.3
control plane version: 1.20.3
data plane version: 1.20.3 (10 proxies)

Additional Information

No response

[howardjohn]

this seems expected, you are configuring the filter twice. you should only configure it once. You can use label selector on the envoy filter to gracefully move over and have only one config at a time

[himanshujaindev]

@howardjohn - We notice 1-2 seconds of downtime during this transition. How can we avoid that? When we attach a label selector to point to a dummy app, and simultaneously, apply pod annotation, downtime is observed.

kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway-unused

[howardjohn]

have 2 deployments one with the annotation and one with the label. don't change running pods

[himanshujaindev]

@howardjohn - If we want to use the EnvoyFilter post 1.20 version as well, will that be an issue?

[howardjohn]

Why would you want to use the Envoyfilter?

either way, you need exactly 1 config per pod

[himanshujaindev]

Due to downtime seen during the transition of Envoyfilter to pod annotation, we cannot move to pod annotation.

We have tried canary deployment, and with this method also we see downtime. https://github.com/istio/istio/issues/46052

[howardjohn]

Can you post the steps you followed? It does not need to be complex.

1. Label all the gateway pods (by changing the deployment) with use-envoyfilter-proxy-protocol: true or something
2. Once that rolls out, change the envoy filter to select use-envoyfilter-proxy-protocol: true. This will be zero downtime
3. Update the deployment, removing the label and adding the annotation

[hzxuzhonghu]

https://github.com/istio/istio/issues/49764

[himanshujaindev]

@howardjohn - The steps mentioned do not cause downtime. Thank you.