IST0158 Warning Despite Matching Control-Plane and Data-Plane Versions

Boenan commented 11 months ago

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

We are encountering an IST0158 warning indicating a version mismatch between the Istio proxy images in the pods and the injection configuration. However, both the control-plane and data-plane are running the same version of Istio, suggesting that this warning might be incorrect or misleading.

Will focus on one deployment (httpbin) as the warning occurs for all deployments.

> kubectl get pods -n httpbin

NAME                       READY   STATUS    RESTARTS   AGE
httpbin-6786dbd4d4-s92bj   2/2     Running   0          14m

Proxy status:

> istioctl proxy-status

NAME                                   CLUSTER      CDS        LDS        EDS        RDS          ECDS         ISTIOD                      VERSION
httpbin-6786dbd4d4-s92bj.httpbin       REDACTED     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-5646874d78-hrp8f     1.19.4

Docker images:

# istiod image (istiod-5646874d78-hrp8f)
company-ecr-repo/pilot:1.19.4

# istio-proxy image (httpbin-6786dbd4d4-s92bj)
company-ecr-repo/proxyv2:1.19.4-distroless

Output when analysing:

> istioctl analyze

Warning [IST0158] (Namespace httpbin) The Istio proxy images of the pods running in the namespace do not match the image defined in the injection configuration (pod names: [httpbin-6786dbd4d4-s92bj httpbin-797587ddc5-j8mwk]). This often happens after upgrading the Istio control-plane and can be fixed by redeploying the pods.

Expected Behavior

No IST0158 warning should be present when the control-plane and data-plane versions match.

Any insights or guidance on this issue would be greatly appreciated. It's causing some confusion as to whether our Istio setup is correctly configured.

Version

> istioctl version
client version: 1.20.0
control plane version: 1.19.4
data plane version: 1.19.4 (12 proxies)

> kubectl version --short

client version: v1.25.5
Server Version: v1.25.15-eks-4f4795d

Additional Information

Running with the following config:

istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: {  }
exclude: { Namespaces: kube-node-lease,kube-public,kube-system,local-path-storage }

CLI version:

version.BuildInfo{Version:"1.20.0", GitRevision:"Homebrew", GolangVersion:"go1.21.4", BuildStatus:"Homebrew", GitTag:"1.20.0"}

The following Istio control plane revisions/versions were found in the cluster: Revision default:

&version.MeshInfo{
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.19.4", GitRevision:"930b482af113506a9aac0f4c3c547aa4f81965bb", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.19.4"},
    },
    {
        Component: "pilot",
        Revision:  "default",
        Info:      version.BuildInfo{Version:"1.19.4", GitRevision:"930b482af113506a9aac0f4c3c547aa4f81965bb", GolangVersion:"", BuildStatus:"Clean", GitTag:"1.19.4"},
    },
}

The following proxy revisions/versions were found in the cluster:

Revision default: Versions {1.19.4}

Fetching logs for the following containers:

httpbin/httpbin/httpbin-6786dbd4d4-s92bj/httpbin
httpbin/httpbin/httpbin-6786dbd4d4-s92bj/istio-proxy

Running Istio analyze on all namespaces and report as below: Analysis Report:

Warning [IST0158] (Namespace httpbin) The Istio proxy images of the pods running in the namespace do not match the image defined in the injection configuration (pod names: [httpbin-6786dbd4d4-s92bj]). This often happens after upgrading the Istio control-plane and can be fixed by redeploying the pods.

zirain commented 11 months ago

can you share the iop configuration? there must be one named installed-state in root namespace(istio-system).

may related to variant.

hanxiaop commented 11 months ago

I think this is expected. How did you inject that distroless proxy image? Looks like it's not injected by the current istiod. You may check the injector configmap to see the image value.

Boenan commented 11 months ago

Istio is installed and managed with Helm so no iop exist.

We are not altering anything with the injector everything is standard.

Looking at the injector configmap we see:

- name: istio-proxy
  image: "{{ .ProxyImage }}"

According to the Official Istio documentation, it's translates to:

${hub}/${image_name}/${tag}-${image_type}

Which is what we expect.

We set:

${hub} → global.hub → company ECR repository
${image_name} → proxyv2 (nothing we override so I guess it default value from global.proxy.image)
${tag} → global.tag → 1.19.4 (default value, nothing we override)
${image_type} → meshConfig.defaultConfig.imageType → distroless

company-ecr-rep/proxyv2:1.19.4-distroless

The generated image value is correct and it's working with no issue but we still see the warning message. If this is expected then the warning message is a bug?

hanxiaop commented 11 months ago

Istio is installed and managed with Helm so no iop exist.

We are not altering anything with the injector everything is standard.

Looking at the injector configmap we see:
- name: istio-proxy
  image: "{{ .ProxyImage }}"
According to the Official Istio documentation, it's translates to:

${hub}/${image_name}/${tag}-${image_type}

Which is what we expect.

We set:

${hub} → global.hub → company ECR repository

${image_name} → proxyv2 (nothing we override so I guess it default value from global.proxy.image)

${tag} → global.tag → 1.19.4 (default value, nothing we override)

${image_type} → meshConfig.defaultConfig.imageType → distroless

company-ecr-rep/proxyv2:1.19.4-distroless

The generated image value is correct and it's working with no issue but we still see the warning message. If this is expected then the warning message is a bug?

Thanks for sharing the details. I guess there's a bug in the analysis message; I'll look into it.

istio / istio