Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

We are following Istio Canary Upgrades and found istioctl uninstall will unexpected remove the revision-tag created for canary upgrades.

Version

$ istioctl version
client version: 1.18.1
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)

$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.6
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

Additional Information

How to repro

Install Istio 1-18-1

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.18.1 sh -

$ istioctl version
no ready Istio pods in "istio-system"
1.18.1

$ istioctl operator init --revision 1-18-1 --tag "1.18.1-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.18.1-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.18.1-distroless
  revision: 1-18-1
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator.yaml
istiooperator.install.istio.io/istiocontrolplane created

$ k label namespace default istio.io/rev=prod-stable --overwrite
namespace/default labeled

$ istioctl tag set prod-canary --revision 1-18-1
Revision tag "prod-canary" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ istioctl tag set prod-stable --revision 1-18-1
Revision tag "prod-stable" created, referencing control plane revision "1-18-1". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-18-1
prod-stable 1-18-1   default

$ k get MutatingWebhookConfigurations
NAME                               WEBHOOKS   AGE
istio-revision-tag-prod-canary     2          27s
istio-revision-tag-prod-stable     2          18s
istio-sidecar-injector-1-18-1      2          13m

Upgrade to istio 1-19-3

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.3 sh -

$ istioctl version
client version: 1.19.3
control plane version: 1.18.1
data plane version: 1.18.1 (1 proxies)

$ istioctl operator init --revision 1-19-3 --tag "1.19.3-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator-1.19.3.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.19.3-distroless
  revision: 1-19-3
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator-1.19.3.yaml
istiooperator.install.istio.io/istiocontrolplane configured

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag set prod-canary --revision 1-19-3 --overwrite
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3
prod-stable 1-19-3   default

Uninstall Istio 1-18-1

$ istioctl uninstall --revision 1-18-1
  Removed Deployment:istio-operator:istio-operator-1-18-1.
  Removed Deployment:istio-system:istiod-1-18-1.
  Removed Service:istio-operator:istio-operator-1-18-1.
  Removed Service:istio-system:istiod-1-18-1.
  Removed ConfigMap:istio-system:istio-1-18-1.
  Removed ConfigMap:istio-system:istio-sidecar-injector-1-18-1.
object: Pod:istio-system:istiod-1-18-1-d4b874889-mbdqx is not being deleted because it no longer exists
  Removed ServiceAccount:istio-operator:istio-operator-1-18-1.
  Removed ServiceAccount:istio-system:istiod-1-18-1.
  Removed RoleBinding:istio-system:istiod-1-18-1.
  Removed Role:istio-system:istiod-1-18-1.
  Removed HorizontalPodAutoscaler:istio-system:istiod-1-18-1.
  Removed PodDisruptionBudget:istio-system:istiod-1-18-1.
  Removed MutatingWebhookConfiguration::istio-sidecar-injector-1-18-1.
  Removed ValidatingWebhookConfiguration::istio-validator-1-18-1-istio-system.
  Removed ClusterRole::istio-operator-1-18-1.
  Removed ClusterRole::istio-reader-clusterrole-1-18-1-istio-system.
  Removed ClusterRole::istiod-clusterrole-1-18-1-istio-system.
  Removed ClusterRole::istiod-gateway-controller-1-18-1-istio-system.
  Removed ClusterRoleBinding::istio-operator-1-18-1.
  Removed ClusterRoleBinding::istio-reader-clusterrole-1-18-1-istio-system.
  Removed ClusterRoleBinding::istiod-clusterrole-1-18-1-istio-system.
  Removed ClusterRoleBinding::istiod-gateway-controller-1-18-1-istio-system.

After above command run, istio-operator log shows istio-operator deleted the revision-tag

Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary. Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.

✔ Ingress gateways installed
- Pruning removed resources[controller-runtime] log.SetLogger(...) was never called; logs will not be displayed.
Detected at:
    >  goroutine 411 [running]:
    >  runtime/debug.Stack()
    >   runtime/debug/stack.go:24 +0x5e
    >  sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/log.go:60 +0xcd
    >  sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc001531780, 0x0?)
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/log/deleg.go:111 +0x32
    >  github.com/go-logr/logr.Logger.Enabled(...)
    >   github.com/go-logr/logr@v1.2.4/logr.go:261
    >  github.com/go-logr/logr.Logger.Info({{0x3449f28?, 0xc001531780?}, 0xc000eb5ad0?}, {0x2eaaa40, 0x14}, {0x0, 0x0, 0x0})
    >   github.com/go-logr/logr@v1.2.4/logr.go:274 +0x72
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0}, {0x2a90f80?, 0xc0000b7bc0?})
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:344 +0x4f7
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000c16be0, {0x3443ec8, 0xc000ae5ef0})
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:266 +0x1c9
    >  sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:227 +0x79
    >  created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 73
    >   sigs.k8s.io/controller-runtime@v0.16.0/pkg/internal/controller/controller.go:223 +0x565
2023-10-18T05:59:43.118237Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.118344Z info    installer   Reconciling IstioOperator
2023-10-18T05:59:43.172465Z info    installer   Updating IstioOperator
2023-10-18T05:59:43.172841Z info    installer   Detecting third-party JWT support
2023-10-18T05:59:43.663698Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:43.964206Z info    installer   Watching a change for istio resource: istio-system/istio-1-18-1
2023-10-18T05:59:44.276378Z info    installer   Watching a change for istio resource: istio-system/istio-sidecar-injector-1-18-1
2023-10-18T05:59:44.784728Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T05:59:44.784764Z info    kube    Pilot K8S CRD controller synced in 43.103µs controller=analysis-controller
2023-10-18T05:59:44.784780Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T05:59:44.784789Z info    kube    Pilot K8S CRD controller synced in 11.401µs controller=analysis-controller
2023-10-18T05:59:44.787734Z info    installer   Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T05:59:44.787781Z info    installer   Pilot is waiting on dependency...
2023-10-18T05:59:44.787797Z info    installer   Cni is waiting on dependency...
2023-10-18T05:59:44.787807Z info    installer   IngressGateways is waiting on dependency...
2023-10-18T05:59:44.787815Z info    installer   Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T05:59:44.787868Z info    installer   Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T05:59:44.787879Z info    installer   EgressGateways is waiting on dependency...
2023-10-18T05:59:44.787744Z info    installer   Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T05:59:44.788452Z info    installer   Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T05:59:45.064198Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.368543Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.524126Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:45.864460Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.055691Z info    installer   Watching a change for istio resource: istio-system/istiod-1-18-1
2023-10-18T05:59:46.074125Z info    installer   Generated manifest objects are the same as cached for component Base.
2023-10-18T05:59:46.074185Z info    installer   Unblocking dependency Pilot.
2023-10-18T05:59:46.074224Z info    installer   Dependency for Pilot has completed, proceeding.
2023-10-18T05:59:46.074234Z info    installer   Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T05:59:46.269846Z info    installer   Generated manifest objects are the same as cached for component Pilot.
2023-10-18T05:59:46.270075Z info    installer   Unblocking dependency Cni.
2023-10-18T05:59:46.270149Z info    installer   Unblocking dependency IngressGateways.
2023-10-18T05:59:46.270190Z info    installer   Unblocking dependency EgressGateways.
2023-10-18T05:59:46.363699Z info    installer   Dependency for EgressGateways has completed, proceeding.
2023-10-18T05:59:46.363748Z info    installer   Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.363867Z info    installer   Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T05:59:46.363883Z info    installer   Dependency for Cni has completed, proceeding.
2023-10-18T05:59:46.363900Z info    installer   Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T05:59:46.363942Z info    installer   Generated manifest objects are the same as cached for component Cni.
2023-10-18T05:59:46.363948Z info    installer   Dependency for IngressGateways has completed, proceeding.
2023-10-18T05:59:46.363957Z info    installer   Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T05:59:46.367348Z info    installer   Watching a change for istio resource: /istio-sidecar-injector-1-18-1
2023-10-18T05:59:46.375135Z info    installer   Generated manifest objects are the same as cached for component IngressGateways.

2023-10-18T05:59:46.764964Z info    installer   Watching a change for istio resource: /istio-validator-1-18-1-istio-system
2023-10-18T05:59:47.166836Z info    installer   Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.383171Z info    installer   Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:47.672115Z info    installer   Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T05:59:48.178424Z info    installer   Watching a change for istio resource: /istio-reader-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.568199Z info    installer   Watching a change for istio resource: /istiod-clusterrole-1-18-1-istio-system
2023-10-18T05:59:48.709472Z info    installer   Watching a change for istio resource: /istiod-gateway-controller-1-18-1-istio-system
2023-10-18T06:00:01.971533Z info    installer   Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.971583Z info      Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:00:01.973419Z info    installer   Watching a change for istio resource: /istio-revision-tag-prod-canary
2023-10-18T06:00:01.978272Z info    installer   Watching a change for istio resource: /istio-revision-tag-prod-stable
2023-10-18T06:00:01.978330Z info    installer   Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-stable from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:01.978365Z info      Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
2023-10-18T06:00:04.586509Z info    installer   Reconciling IstioOperator
2023-10-18T06:00:04.680744Z info    installer   Updating IstioOperator
2023-10-18T06:00:04.680811Z info    installer   Detecting third-party JWT support
2023-10-18T06:00:06.371940Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:00:06.371984Z info    kube    Pilot K8S CRD controller synced in 51.203µs controller=analysis-controller
2023-10-18T06:00:06.372005Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:00:06.372015Z info    kube    Pilot K8S CRD controller synced in 14.2µs   controller=analysis-controller
2023-10-18T06:00:06.374930Z info    installer   Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:00:06.375007Z info    installer   Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:00:06.375030Z info    installer   IngressGateways is waiting on dependency...
2023-10-18T06:00:06.375042Z info    installer   EgressGateways is waiting on dependency...
2023-10-18T06:00:06.375050Z info    installer   Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:00:06.467573Z info    installer   Cni is waiting on dependency...
2023-10-18T06:00:06.467621Z info    installer   Pilot is waiting on dependency...
2023-10-18T06:00:06.467644Z info    installer   Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:00:06.467706Z info    installer   Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:00:07.473118Z info    installer   Generated manifest objects are the same as cached for component Base.
2023-10-18T06:00:07.473176Z info    installer   Unblocking dependency Pilot.
2023-10-18T06:00:07.473196Z info    installer   Dependency for Pilot has completed, proceeding.
2023-10-18T06:00:07.473206Z info    installer   Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
2023-10-18T06:00:07.765838Z info    installer   Generated manifest objects are the same as cached for component Pilot.
2023-10-18T06:00:07.765881Z info    installer   Unblocking dependency Cni.
2023-10-18T06:00:07.765887Z info    installer   Unblocking dependency IngressGateways.
2023-10-18T06:00:07.765891Z info    installer   Unblocking dependency EgressGateways.
2023-10-18T06:00:07.765898Z info    installer   Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:00:07.765908Z info    installer   Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.765953Z info    installer   Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:00:07.765959Z info    installer   Dependency for Cni has completed, proceeding.
2023-10-18T06:00:07.765964Z info    installer   Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:00:07.765989Z info    installer   Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:00:07.765994Z info    installer   Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:00:07.765998Z info    installer   Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:00:07.772990Z info    installer   Generated manifest objects are the same as cached for component IngressGateways.
- Pruning removed resources

$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list

@hanxiaop could you help triage this issue as you are recently committer for this file https://github.com/istio/istio/commits/master/operator/cmd/mesh/uninstall.go

@sonnyhcl Have you set tag after installing 1-19-3?

@hanxiaop sorry for the missing. We did re-tag 1-19-3 in our prod environment, but missed in above re-pro context. Actually the deletion happened inside istio-opereator as log shows. I've update the re-pro steps and logs

Found a simpler way to re-pro. Tried 1.19.0/1.19.1/1.19.3. It seems at least istio-operator >= 1.19.0 will always clean MutatingWebhookConfiguration::istio-revision-tag, which breaks all canary-upgrade scenarios.

Install Istio 1.19.0

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.0 sh -

$ istioctl version
no ready Istio pods in "istio-system"
client version: 1.19.0

$ istioctl operator init --revision 1-19-0 --tag "1.19.0-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator-1.19.0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.19.0-distroless
  revision: 1-19-0
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator-1.19.0.yaml
istiooperator.install.istio.io/istiocontrolplane configured

$ istioctl tag set prod-stable --revision 1-19-0 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-0". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3

Restart istio-operator

$ k rollout restart -n istio-system istio-operator-1-19-0

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3

wait 1min to let istio-operator reconciling

2023-10-18T06:36:28.336109Z info    ControlZ available at 127.0.0.1:9876
2023-10-18T06:36:28.336216Z info    Leader election cm: istio-operator-lock-1-19-0
2023-10-18T06:36:28.336564Z info    Creating operator metrics exporter available at 127.0.0.1:15014
2023-10-18T06:36:28.336962Z info    Registering Components.
2023-10-18T06:36:28.337615Z info    installer   Adding controller for IstioOperator.
2023-10-18T06:36:28.363785Z info    installer   Controller added
2023-10-18T06:36:28.363814Z info    Starting the Cmd.
2023-10-18T06:36:28.364966Z info    klog    attempting to acquire leader lease istio-operator/istio-operator-lock-1-19-0...
2023-10-18T06:36:51.172669Z info    klog    successfully acquired lease istio-operator/istio-operator-lock-1-19-0
2023-10-18T06:36:52.037423Z info    installer   Reconciling IstioOperator
2023-10-18T06:36:52.127816Z info    installer   Updating IstioOperator
2023-10-18T06:36:52.128074Z info    installer   Detecting third-party JWT support
2023-10-18T06:36:53.745097Z info    cluster "" kube client started
2023-10-18T06:36:53.845793Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:36:53.845882Z info    kube    Pilot K8S CRD controller synced in 96.702µs controller=analysis-controller
2023-10-18T06:36:53.845919Z info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:36:53.845933Z info    kube    Pilot K8S CRD controller synced in 16.4µs   controller=analysis-controller
2023-10-18T06:36:53.848672Z info    installer   Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:36:53.848689Z info    installer   Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:36:53.848847Z info    installer   Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:36:53.848946Z info    installer   Pilot is waiting on dependency...
2023-10-18T06:36:53.848963Z info    installer   Cni is waiting on dependency...
2023-10-18T06:36:53.848974Z info    installer   Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:36:53.849110Z info    installer   Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:36:53.849130Z info    installer   IngressGateways is waiting on dependency...
2023-10-18T06:36:53.849137Z info    installer   EgressGateways is waiting on dependency...
2023-10-18T06:36:54.533033Z info    installer   The following objects differ between generated manifest and cache: 
 - ServiceAccount:istio-system:istio-reader-service-account
 - CustomResourceDefinition::wasmplugins.extensions.istio.io
 - CustomResourceDefinition::destinationrules.networking.istio.io
 - CustomResourceDefinition::envoyfilters.networking.istio.io
 - CustomResourceDefinition::gateways.networking.istio.io
 - CustomResourceDefinition::proxyconfigs.networking.istio.io
 - CustomResourceDefinition::serviceentries.networking.istio.io
 - CustomResourceDefinition::sidecars.networking.istio.io
 - CustomResourceDefinition::virtualservices.networking.istio.io
 - CustomResourceDefinition::workloadentries.networking.istio.io
 - CustomResourceDefinition::workloadgroups.networking.istio.io
 - CustomResourceDefinition::authorizationpolicies.security.istio.io
 - CustomResourceDefinition::peerauthentications.security.istio.io
 - CustomResourceDefinition::requestauthentications.security.istio.io
 - CustomResourceDefinition::telemetries.telemetry.istio.io
 - CustomResourceDefinition::istiooperators.install.istio.io
2023-10-18T06:36:54.533164Z info    installer   using server side apply to update obj: CustomResourceDefinition//authorizationpolicies.security.istio.io

2023-10-18T06:36:54.555213Z info    installer   using server side apply to update obj: CustomResourceDefinition//destinationrules.networking.istio.io
2023-10-18T06:36:54.742451Z info    installer   using server side apply to update obj: CustomResourceDefinition//envoyfilters.networking.istio.io
2023-10-18T06:36:54.762292Z info    installer   using server side apply to update obj: CustomResourceDefinition//gateways.networking.istio.io
2023-10-18T06:36:54.781690Z info    installer   using server side apply to update obj: CustomResourceDefinition//istiooperators.install.istio.io
2023-10-18T06:36:54.791087Z info    installer   using server side apply to update obj: CustomResourceDefinition//peerauthentications.security.istio.io
2023-10-18T06:36:54.805337Z info    installer   using server side apply to update obj: CustomResourceDefinition//proxyconfigs.networking.istio.io
2023-10-18T06:36:54.815990Z info    installer   using server side apply to update obj: CustomResourceDefinition//requestauthentications.security.istio.io
2023-10-18T06:36:54.837951Z info    installer   using server side apply to update obj: CustomResourceDefinition//serviceentries.networking.istio.io
2023-10-18T06:36:54.859512Z info    installer   using server side apply to update obj: CustomResourceDefinition//sidecars.networking.istio.io
2023-10-18T06:36:54.882910Z info    installer   using server side apply to update obj: CustomResourceDefinition//telemetries.telemetry.istio.io
2023-10-18T06:36:54.901104Z info    installer   using server side apply to update obj: CustomResourceDefinition//virtualservices.networking.istio.io
2023-10-18T06:36:54.994491Z info    installer   using server side apply to update obj: CustomResourceDefinition//wasmplugins.extensions.istio.io
2023-10-18T06:36:55.007659Z info    installer   using server side apply to update obj: CustomResourceDefinition//workloadentries.networking.istio.io
2023-10-18T06:36:55.019101Z info    installer   using server side apply to update obj: CustomResourceDefinition//workloadgroups.networking.istio.io
2023-10-18T06:36:55.042370Z info    installer   using server side apply to update obj: ServiceAccount/istio-system/istio-reader-service-account
2023-10-18T06:36:55.557427Z info    installer   established CRD authorizationpolicies.security.istio.io
2023-10-18T06:36:55.570827Z info    installer   established CRD destinationrules.networking.istio.io
2023-10-18T06:36:55.577036Z info    installer   established CRD envoyfilters.networking.istio.io
2023-10-18T06:36:55.584606Z info    installer   established CRD gateways.networking.istio.io
2023-10-18T06:36:55.589211Z info    installer   established CRD istiooperators.install.istio.io
2023-10-18T06:36:55.594265Z info    installer   established CRD peerauthentications.security.istio.io
2023-10-18T06:36:55.599570Z info    installer   established CRD proxyconfigs.networking.istio.io
2023-10-18T06:36:55.605172Z info    installer   established CRD requestauthentications.security.istio.io
2023-10-18T06:36:55.610211Z info    installer   established CRD serviceentries.networking.istio.io
2023-10-18T06:36:55.615857Z info    installer   established CRD sidecars.networking.istio.io
2023-10-18T06:36:55.620660Z info    installer   established CRD telemetries.telemetry.istio.io
2023-10-18T06:36:55.628498Z info    installer   established CRD virtualservices.networking.istio.io
2023-10-18T06:36:55.634131Z info    installer   established CRD wasmplugins.extensions.istio.io
2023-10-18T06:36:55.638900Z info    installer   established CRD workloadentries.networking.istio.io
2023-10-18T06:36:55.643886Z info    installer   established CRD workloadgroups.networking.istio.io
2023-10-18T06:36:55.643907Z info    installer   Finished applying CRDs.
2023-10-18T06:36:55.644125Z info    installer   Unblocking dependency Pilot.
2023-10-18T06:36:55.644145Z info    installer   Dependency for Pilot has completed, proceeding.
2023-10-18T06:36:55.644155Z info    installer   Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
- Processing resources for Istio core.
2023-10-18T06:36:55.738768Z info    installer   The following objects differ between generated manifest and cache: 
 - HorizontalPodAutoscaler:istio-system:istiod-1-19-0
 - ClusterRole::istiod-clusterrole-1-19-0-istio-system
 - ClusterRole::istiod-gateway-controller-1-19-0-istio-system
 - ClusterRoleBinding::istiod-clusterrole-1-19-0-istio-system
 - ClusterRoleBinding::istiod-gateway-controller-1-19-0-istio-system
 - ConfigMap:istio-system:istio-1-19-0
 - Deployment:istio-system:istiod-1-19-0
 - ConfigMap:istio-system:istio-sidecar-injector-1-19-0
 - MutatingWebhookConfiguration::istio-sidecar-injector-1-19-0
 - PodDisruptionBudget:istio-system:istiod-1-19-0
 - ClusterRole::istio-reader-clusterrole-1-19-0-istio-system
 - ClusterRoleBinding::istio-reader-clusterrole-1-19-0-istio-system
 - Role:istio-system:istiod-1-19-0
 - RoleBinding:istio-system:istiod-1-19-0
 - Service:istio-system:istiod-1-19-0
 - ServiceAccount:istio-system:istiod-1-19-0
 - ValidatingWebhookConfiguration::istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.738846Z info    installer   using server side apply to update obj: ServiceAccount/istio-system/istiod-1-19-0
2023-10-18T06:36:55.834335Z info    installer   using server side apply to update obj: ClusterRole//istio-reader-clusterrole-1-19-0-istio-system
✔ Istio core installed
2023-10-18T06:36:55.845991Z info    installer   using server side apply to update obj: ClusterRole//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.854131Z info    installer   using server side apply to update obj: ClusterRole//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.860114Z info    installer   using server side apply to update obj: ClusterRoleBinding//istio-reader-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.866759Z info    installer   using server side apply to update obj: ClusterRoleBinding//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.873611Z info    installer   using server side apply to update obj: ClusterRoleBinding//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.880284Z info    installer   using server side apply to update obj: ValidatingWebhookConfiguration//istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.889650Z info    installer   using server side apply to update obj: ConfigMap/istio-system/istio-1-19-0
2023-10-18T06:36:55.903834Z info    installer   using server side apply to update obj: ConfigMap/istio-system/istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.929110Z info    installer   using server side apply to update obj: MutatingWebhookConfiguration//istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.940177Z info    installer   using server side apply to update obj: Deployment/istio-system/istiod-1-19-0
2023-10-18T06:36:55.957152Z info    installer   using server side apply to update obj: PodDisruptionBudget/istio-system/istiod-1-19-0
2023-10-18T06:36:55.964597Z info    installer   using server side apply to update obj: Role/istio-system/istiod-1-19-0
2023-10-18T06:36:55.971591Z info    installer   using server side apply to update obj: RoleBinding/istio-system/istiod-1-19-0
2023-10-18T06:36:55.977989Z info    installer   using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istiod-1-19-0
2023-10-18T06:36:55.985071Z info    installer   using server side apply to update obj: Service/istio-system/istiod-1-19-0
- Processing resources for Istiod.
2023-10-18T06:36:56.007220Z info    installer   Unblocking dependency Cni.
2023-10-18T06:36:56.007317Z info    installer   Unblocking dependency IngressGateways.
2023-10-18T06:36:56.007324Z info    installer   Unblocking dependency EgressGateways.
2023-10-18T06:36:56.007332Z info    installer   Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:36:56.007361Z info    installer   Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.007547Z info    installer   Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:36:56.007582Z info    installer   Dependency for Cni has completed, proceeding.
2023-10-18T06:36:56.007594Z info    installer   Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:36:56.007620Z info    installer   Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:36:56.007625Z info    installer   Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:36:56.007629Z info    installer   Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.012152Z info    installer   The following objects differ between generated manifest and cache: 
 - HorizontalPodAutoscaler:istio-system:istio-ingressgateway
 - Deployment:istio-system:istio-ingressgateway
 - PodDisruptionBudget:istio-system:istio-ingressgateway
 - Role:istio-system:istio-ingressgateway-sds
 - RoleBinding:istio-system:istio-ingressgateway-sds
 - Service:istio-system:istio-ingressgateway
 - ServiceAccount:istio-system:istio-ingressgateway-service-account
2023-10-18T06:36:56.012237Z info    installer   using server side apply to update obj: ServiceAccount/istio-system/istio-ingressgateway-service-account
✔ Istiod installed
2023-10-18T06:36:56.018353Z info    installer   using server side apply to update obj: Deployment/istio-system/istio-ingressgateway
2023-10-18T06:36:56.042772Z info    installer   using server side apply to update obj: PodDisruptionBudget/istio-system/istio-ingressgateway
2023-10-18T06:36:56.050471Z info    installer   using server side apply to update obj: Role/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.057116Z info    installer   using server side apply to update obj: RoleBinding/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.064533Z info    installer   using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istio-ingressgateway
2023-10-18T06:36:56.072139Z info    installer   using server side apply to update obj: Service/istio-system/istio-ingressgateway
- Processing resources for Ingress gateways.
✔ Ingress gateways installed
2023-10-18T06:37:11.735610Z info    installer   Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736122Z info      Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736279Z info    installer   Watching a change for istio resource: /istio-revision-tag-prod-canary

Now istio revision tag has been cleaned.

$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list

Install Istio 1.19.0

$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.0 sh -

$ istioctl version
no ready Istio pods in "istio-system"
client version: 1.19.0

$ istioctl operator init --revision 1-19-0 --tag "1.19.0-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete

$ cat istio-operator-1.19.0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istiocontrolplane
spec:
  profile: minimal
  tag: 1.19.0-distroless
  revision: 1-19-0
  components:
    pilot:
      enabled: true
    ingressGateways:
      - enabled: true
        name: istio-ingressgateway

$ k apply -f istio-operator-1.19.0.yaml
istiooperator.install.istio.io/istiocontrolplane configured

$ istioctl tag set prod-stable --revision 1-19-0 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-0". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3

Restart istio-operator

$ k rollout restart -n istio-system istio-operator-1-19-0

$ istioctl tag list
TAG         REVISION NAMESPACES
prod-canary 1-19-3

wait 1min to let istio-operator reconciling

2023-10-18T06:36:28.336109Z   info    ControlZ available at 127.0.0.1:9876
2023-10-18T06:36:28.336216Z   info    Leader election cm: istio-operator-lock-1-19-0
2023-10-18T06:36:28.336564Z   info    Creating operator metrics exporter available at 127.0.0.1:15014
2023-10-18T06:36:28.336962Z   info    Registering Components.
2023-10-18T06:36:28.337615Z   info    installer   Adding controller for IstioOperator.
2023-10-18T06:36:28.363785Z   info    installer   Controller added
2023-10-18T06:36:28.363814Z   info    Starting the Cmd.
2023-10-18T06:36:28.364966Z   info    klog    attempting to acquire leader lease istio-operator/istio-operator-lock-1-19-0...
2023-10-18T06:36:51.172669Z   info    klog    successfully acquired lease istio-operator/istio-operator-lock-1-19-0
2023-10-18T06:36:52.037423Z   info    installer   Reconciling IstioOperator
2023-10-18T06:36:52.127816Z   info    installer   Updating IstioOperator
2023-10-18T06:36:52.128074Z   info    installer   Detecting third-party JWT support
2023-10-18T06:36:53.745097Z   info    cluster "" kube client started
2023-10-18T06:36:53.845793Z   info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:36:53.845882Z   info    kube    Pilot K8S CRD controller synced in 96.702µs controller=analysis-controller
2023-10-18T06:36:53.845919Z   info    kube    Starting Pilot K8S CRD controller   controller=analysis-controller
2023-10-18T06:36:53.845933Z   info    kube    Pilot K8S CRD controller synced in 16.4µs   controller=analysis-controller
2023-10-18T06:36:53.848672Z   info    installer   Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:36:53.848689Z   info    installer   Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:36:53.848847Z   info    installer   Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:36:53.848946Z   info    installer   Pilot is waiting on dependency...
2023-10-18T06:36:53.848963Z   info    installer   Cni is waiting on dependency...
2023-10-18T06:36:53.848974Z   info    installer   Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:36:53.849110Z   info    installer   Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:36:53.849130Z   info    installer   IngressGateways is waiting on dependency...
2023-10-18T06:36:53.849137Z   info    installer   EgressGateways is waiting on dependency...
2023-10-18T06:36:54.533033Z   info    installer   The following objects differ between generated manifest and cache: 
 - ServiceAccount:istio-system:istio-reader-service-account
 - CustomResourceDefinition::wasmplugins.extensions.istio.io
 - CustomResourceDefinition::destinationrules.networking.istio.io
 - CustomResourceDefinition::envoyfilters.networking.istio.io
 - CustomResourceDefinition::gateways.networking.istio.io
 - CustomResourceDefinition::proxyconfigs.networking.istio.io
 - CustomResourceDefinition::serviceentries.networking.istio.io
 - CustomResourceDefinition::sidecars.networking.istio.io
 - CustomResourceDefinition::virtualservices.networking.istio.io
 - CustomResourceDefinition::workloadentries.networking.istio.io
 - CustomResourceDefinition::workloadgroups.networking.istio.io
 - CustomResourceDefinition::authorizationpolicies.security.istio.io
 - CustomResourceDefinition::peerauthentications.security.istio.io
 - CustomResourceDefinition::requestauthentications.security.istio.io
 - CustomResourceDefinition::telemetries.telemetry.istio.io
 - CustomResourceDefinition::istiooperators.install.istio.io
2023-10-18T06:36:54.533164Z   info    installer   using server side apply to update obj: CustomResourceDefinition//authorizationpolicies.security.istio.io

2023-10-18T06:36:54.555213Z   info    installer   using server side apply to update obj: CustomResourceDefinition//destinationrules.networking.istio.io
2023-10-18T06:36:54.742451Z   info    installer   using server side apply to update obj: CustomResourceDefinition//envoyfilters.networking.istio.io
2023-10-18T06:36:54.762292Z   info    installer   using server side apply to update obj: CustomResourceDefinition//gateways.networking.istio.io
2023-10-18T06:36:54.781690Z   info    installer   using server side apply to update obj: CustomResourceDefinition//istiooperators.install.istio.io
2023-10-18T06:36:54.791087Z   info    installer   using server side apply to update obj: CustomResourceDefinition//peerauthentications.security.istio.io
2023-10-18T06:36:54.805337Z   info    installer   using server side apply to update obj: CustomResourceDefinition//proxyconfigs.networking.istio.io
2023-10-18T06:36:54.815990Z   info    installer   using server side apply to update obj: CustomResourceDefinition//requestauthentications.security.istio.io
2023-10-18T06:36:54.837951Z   info    installer   using server side apply to update obj: CustomResourceDefinition//serviceentries.networking.istio.io
2023-10-18T06:36:54.859512Z   info    installer   using server side apply to update obj: CustomResourceDefinition//sidecars.networking.istio.io
2023-10-18T06:36:54.882910Z   info    installer   using server side apply to update obj: CustomResourceDefinition//telemetries.telemetry.istio.io
2023-10-18T06:36:54.901104Z   info    installer   using server side apply to update obj: CustomResourceDefinition//virtualservices.networking.istio.io
2023-10-18T06:36:54.994491Z   info    installer   using server side apply to update obj: CustomResourceDefinition//wasmplugins.extensions.istio.io
2023-10-18T06:36:55.007659Z   info    installer   using server side apply to update obj: CustomResourceDefinition//workloadentries.networking.istio.io
2023-10-18T06:36:55.019101Z   info    installer   using server side apply to update obj: CustomResourceDefinition//workloadgroups.networking.istio.io
2023-10-18T06:36:55.042370Z   info    installer   using server side apply to update obj: ServiceAccount/istio-system/istio-reader-service-account
2023-10-18T06:36:55.557427Z   info    installer   established CRD authorizationpolicies.security.istio.io
2023-10-18T06:36:55.570827Z   info    installer   established CRD destinationrules.networking.istio.io
2023-10-18T06:36:55.577036Z   info    installer   established CRD envoyfilters.networking.istio.io
2023-10-18T06:36:55.584606Z   info    installer   established CRD gateways.networking.istio.io
2023-10-18T06:36:55.589211Z   info    installer   established CRD istiooperators.install.istio.io
2023-10-18T06:36:55.594265Z   info    installer   established CRD peerauthentications.security.istio.io
2023-10-18T06:36:55.599570Z   info    installer   established CRD proxyconfigs.networking.istio.io
2023-10-18T06:36:55.605172Z   info    installer   established CRD requestauthentications.security.istio.io
2023-10-18T06:36:55.610211Z   info    installer   established CRD serviceentries.networking.istio.io
2023-10-18T06:36:55.615857Z   info    installer   established CRD sidecars.networking.istio.io
2023-10-18T06:36:55.620660Z   info    installer   established CRD telemetries.telemetry.istio.io
2023-10-18T06:36:55.628498Z   info    installer   established CRD virtualservices.networking.istio.io
2023-10-18T06:36:55.634131Z   info    installer   established CRD wasmplugins.extensions.istio.io
2023-10-18T06:36:55.638900Z   info    installer   established CRD workloadentries.networking.istio.io
2023-10-18T06:36:55.643886Z   info    installer   established CRD workloadgroups.networking.istio.io
2023-10-18T06:36:55.643907Z   info    installer   Finished applying CRDs.
2023-10-18T06:36:55.644125Z   info    installer   Unblocking dependency Pilot.
2023-10-18T06:36:55.644145Z   info    installer   Dependency for Pilot has completed, proceeding.
2023-10-18T06:36:55.644155Z   info    installer   Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
- Processing resources for Istio core.
2023-10-18T06:36:55.738768Z   info    installer   The following objects differ between generated manifest and cache: 
 - HorizontalPodAutoscaler:istio-system:istiod-1-19-0
 - ClusterRole::istiod-clusterrole-1-19-0-istio-system
 - ClusterRole::istiod-gateway-controller-1-19-0-istio-system
 - ClusterRoleBinding::istiod-clusterrole-1-19-0-istio-system
 - ClusterRoleBinding::istiod-gateway-controller-1-19-0-istio-system
 - ConfigMap:istio-system:istio-1-19-0
 - Deployment:istio-system:istiod-1-19-0
 - ConfigMap:istio-system:istio-sidecar-injector-1-19-0
 - MutatingWebhookConfiguration::istio-sidecar-injector-1-19-0
 - PodDisruptionBudget:istio-system:istiod-1-19-0
 - ClusterRole::istio-reader-clusterrole-1-19-0-istio-system
 - ClusterRoleBinding::istio-reader-clusterrole-1-19-0-istio-system
 - Role:istio-system:istiod-1-19-0
 - RoleBinding:istio-system:istiod-1-19-0
 - Service:istio-system:istiod-1-19-0
 - ServiceAccount:istio-system:istiod-1-19-0
 - ValidatingWebhookConfiguration::istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.738846Z   info    installer   using server side apply to update obj: ServiceAccount/istio-system/istiod-1-19-0
2023-10-18T06:36:55.834335Z   info    installer   using server side apply to update obj: ClusterRole//istio-reader-clusterrole-1-19-0-istio-system
✔ Istio core installed
2023-10-18T06:36:55.845991Z   info    installer   using server side apply to update obj: ClusterRole//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.854131Z   info    installer   using server side apply to update obj: ClusterRole//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.860114Z   info    installer   using server side apply to update obj: ClusterRoleBinding//istio-reader-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.866759Z   info    installer   using server side apply to update obj: ClusterRoleBinding//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.873611Z   info    installer   using server side apply to update obj: ClusterRoleBinding//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.880284Z   info    installer   using server side apply to update obj: ValidatingWebhookConfiguration//istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.889650Z   info    installer   using server side apply to update obj: ConfigMap/istio-system/istio-1-19-0
2023-10-18T06:36:55.903834Z   info    installer   using server side apply to update obj: ConfigMap/istio-system/istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.929110Z   info    installer   using server side apply to update obj: MutatingWebhookConfiguration//istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.940177Z   info    installer   using server side apply to update obj: Deployment/istio-system/istiod-1-19-0
2023-10-18T06:36:55.957152Z   info    installer   using server side apply to update obj: PodDisruptionBudget/istio-system/istiod-1-19-0
2023-10-18T06:36:55.964597Z   info    installer   using server side apply to update obj: Role/istio-system/istiod-1-19-0
2023-10-18T06:36:55.971591Z   info    installer   using server side apply to update obj: RoleBinding/istio-system/istiod-1-19-0
2023-10-18T06:36:55.977989Z   info    installer   using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istiod-1-19-0
2023-10-18T06:36:55.985071Z   info    installer   using server side apply to update obj: Service/istio-system/istiod-1-19-0
- Processing resources for Istiod.
2023-10-18T06:36:56.007220Z   info    installer   Unblocking dependency Cni.
2023-10-18T06:36:56.007317Z   info    installer   Unblocking dependency IngressGateways.
2023-10-18T06:36:56.007324Z   info    installer   Unblocking dependency EgressGateways.
2023-10-18T06:36:56.007332Z   info    installer   Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:36:56.007361Z   info    installer   Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.007547Z   info    installer   Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:36:56.007582Z   info    installer   Dependency for Cni has completed, proceeding.
2023-10-18T06:36:56.007594Z   info    installer   Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:36:56.007620Z   info    installer   Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:36:56.007625Z   info    installer   Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:36:56.007629Z   info    installer   Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.012152Z   info    installer   The following objects differ between generated manifest and cache: 
 - HorizontalPodAutoscaler:istio-system:istio-ingressgateway
 - Deployment:istio-system:istio-ingressgateway
 - PodDisruptionBudget:istio-system:istio-ingressgateway
 - Role:istio-system:istio-ingressgateway-sds
 - RoleBinding:istio-system:istio-ingressgateway-sds
 - Service:istio-system:istio-ingressgateway
 - ServiceAccount:istio-system:istio-ingressgateway-service-account
2023-10-18T06:36:56.012237Z   info    installer   using server side apply to update obj: ServiceAccount/istio-system/istio-ingressgateway-service-account
✔ Istiod installed
2023-10-18T06:36:56.018353Z   info    installer   using server side apply to update obj: Deployment/istio-system/istio-ingressgateway
2023-10-18T06:36:56.042772Z   info    installer   using server side apply to update obj: PodDisruptionBudget/istio-system/istio-ingressgateway
2023-10-18T06:36:56.050471Z   info    installer   using server side apply to update obj: Role/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.057116Z   info    installer   using server side apply to update obj: RoleBinding/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.064533Z   info    installer   using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istio-ingressgateway
2023-10-18T06:36:56.072139Z   info    installer   using server side apply to update obj: Service/istio-system/istio-ingressgateway
- Processing resources for Ingress gateways.
✔ Ingress gateways installed
2023-10-18T06:37:11.735610Z   info    installer   Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736122Z   info      Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736279Z   info    installer   Watching a change for istio resource: /istio-revision-tag-prod-canary

Now istio revision tag has been cleaned.

$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list

@sonnyhcl Thanks for the info! This seems like a regression. I'll look into it.

BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.

We observe a similar issue with istioctl install. Ways to reproduce:

istio 1.19 installed

istioctl tag list
TAG     REVISION NAMESPACES
default 1-19     a,b,c,d,e

Apply the istio 1.19 manifest again with istioctl

✔ Istio core installed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
✔ Istiod installed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
✔ CNI installed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
✔ Ingress gateways installed                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
- Pruning removed resources                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           Removed MutatingWebhookConfiguration::istio-revision-tag-default.
  Removed ValidatingWebhookConfiguration::istiod-default-validator.
✔ Installation complete

istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list

Also observed istioctl install will prune user set revision tag.

2023-10-24 01:55:41.664295: ++ istioctl install -y --verify --revision 1-19-3 -f setup/istio-operator.yaml
2023-10-24 01:55:42.422813: WARNING: Istio is being upgraded from 1.19.0 to 1.19.3.
2023-10-24 01:55:42.423134: Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings.
2023-10-24 01:55:43.364625:
2023-10-24 01:55:45.723057: - Processing resources for Istio core.
2023-10-24 01:55:45.820562: \N{HEAVY CHECK MARK} Istio core installed
2023-10-24 01:55:47.009512: - Processing resources for Istiod.
2023-10-24 01:55:47.069855: \N{HEAVY CHECK MARK} Istiod installed
2023-10-24 01:55:47.467949: - Processing resources for Ingress gateways.
2023-10-24 01:55:47.469677: \N{HEAVY CHECK MARK} Ingress gateways installed
2023-10-24 01:55:52.979047: - Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-24 01:55:53.026674: Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.

This is blocking us upgrading to istio 1.19.3, which is a security fix. @hanxiaop do you know any other workaround? considering it will take time to release a new istio version.

Also observed istioctl install will prune user set revision tag.

2023-10-24 01:55:41.664295: ++ istioctl install -y --verify --revision 1-19-3 -f setup/istio-operator.yaml
2023-10-24 01:55:42.422813: WARNING: Istio is being upgraded from 1.19.0 to 1.19.3.
2023-10-24 01:55:42.423134: Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings.
2023-10-24 01:55:43.364625:
2023-10-24 01:55:45.723057: - Processing resources for Istio core.
2023-10-24 01:55:45.820562: \N{HEAVY CHECK MARK} Istio core installed
2023-10-24 01:55:47.009512: - Processing resources for Istiod.
2023-10-24 01:55:47.069855: \N{HEAVY CHECK MARK} Istiod installed
2023-10-24 01:55:47.467949: - Processing resources for Ingress gateways.
2023-10-24 01:55:47.469677: \N{HEAVY CHECK MARK} Ingress gateways installed
2023-10-24 01:55:52.979047: - Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-24 01:55:53.026674: Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.

This is blocking us upgrading to istio 1.19.3, which is a security fix. @hanxiaop do you know any other workaround? considering it will take time to release a new istio version.

@sonnyhcl I'm going to fix it up soon. Before having the fix, maybe you can try to set the annotation install.operator.istio.io/owning-resource-not-pruned="true" to keep the resource? If it's not working, deleting the operator related labels (starting with install.operator.istio.io) of the tag webhook you've created with istioctl.

both workarounds are fine, with small diff.

~annotate~ label `install.operator.istio.io/owning-resource-not-pruned=true` works, and the labels won't be overwrite by using `istioctl tag set` again.

label prod-stable thus won't be pruned

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag set prod-canary --revision 1-19-3
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ k get MutatingWebhookConfigurations
NAME                               WEBHOOKS   AGE
istio-revision-tag-prod-canary     2          100s
istio-revision-tag-prod-stable     2          97s
istio-sidecar-injector-1-19-3      2          4d1h

$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-not-pruned=true --overwrite
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable labeled

$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
    install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
    install.operator.istio.io/owning-resource-namespace: istio-system
    install.operator.istio.io/owning-resource-not-pruned: "true"

$ istioctl install -y --revision 1-19-3 --verify -f istio-operator-1.19.3.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
- Pruning removed resources
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
✔ Installation complete

owning-resource-not-pruned label still exists

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
    install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
    install.operator.istio.io/owning-resource-namespace: istio-system
    install.operator.istio.io/owning-resource-not-pruned: "true"

delete operator related labels also works, but the labels will come back once using `istioctl tag set` again.

clean prod-stable labels thus won't be pruned

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ istioctl tag set prod-canary --revision 1-19-3 --overwrite
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'

$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-namespace-
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable unlabeled

$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable unlabeled

$ istioctl install -y --revision 1-19-3 --verify -f istio-operator-1.19.3.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
- Pruning removed resources
  Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
✔ Installation complete

label is overwritten once `istioctl install` again

$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
    install.operator.istio.io/owning-resource-not-pruned: "true"

$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'

$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
    install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
    install.operator.istio.io/owning-resource-namespace: istio-system
    install.operator.istio.io/owning-resource-not-pruned: "true"

@sonnyhcl Thanks for the verification, I've submitted a fix and I think it can be included in the next release.

BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.

Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?

BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.

Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?

I mean the in-cluster operator controller is not encouraged, which is deployed by istioctl operator init. Currently we encourage people to use Helm, or istioctl to install Istio.

BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.

Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?

I mean the in-cluster operator controller is not encouraged, which is deployed by istioctl operator init. Currently we encourage people to use Helm, or istioctl to install Istio.

Thanks for clarification. Now I understand the difference. But somehow myself and my team likes that in-cluster operator and we are struck with it. If we have to move from that to "helm" it is big work for us.

Anyways, I hit with a bug on istioctl binary of 1.18.5 version. If I run the below command, i get an error

istioctl tag set stable --revision 1-18-5 --overwrite
Error: failed to apply tag webhook MutatingWebhookConfiguration to cluster: failed to apply tag manifests to cluster: failed applying manifest /var/folders/yb/7gsxhlrn7t97nnrh5b565bhdscxz7p/T/revision-tag-manifest-3851079461: error validating "/var/folders/yb/7gsxhlrn7t97nnrh5b565bhdscxz7p/T/revision-tag-manifest-3851079461": error validating data: the server has asked for the client to provide credentials; if you choose to ignore these errors, turn validation off with --validate=false:

But same command with 1.19.3 binary, it is successful.

May I know is the fix (#47538) included in the 1.19.4 release train or not? @hanxiaop

May I know is the fix (#47538) included in the 1.19.4 release train or not? @hanxiaop

Looks like it's not. Will be included in the next release.

Hi @hanxiaop it looks like it was released with 1.19.5.

Unfortunately, we still see this issue. I described in https://github.com/istio/istio/issues/47423#issuecomment-1774482179 how to reproduce.

Hi @hanxiaop it looks like it was released with 1.19.5.

Unfortunately, we still see this issue. I described in #47423 (comment) how to reproduce.

@wiegandf Sent the wrong fix, which was not actually working, and the release note was included. I've already fixed this recently, and the fix will be available in 1.19.7 and 1.20.3.

If it's not working, deleting the operator related labels (starting with install.operator.istio.io) of the tag webhook you've created with istioctl.

@wiegandf This can be a workaround before waiting for the new release.

istio / istio

istio-operator/istioctl > 1.19 unexpectedly remove MutatingWebhookConfiguration::istio-revision-tag #47423

Is this the right place to submit this?

Bug Description

Version

Additional Information

How to repro

Install Istio 1-18-1

Upgrade to istio 1-19-3

Uninstall Istio 1-18-1

Install Istio 1.19.0

Restart istio-operator

Install Istio 1.19.0

Restart istio-operator

~annotate~ label `install.operator.istio.io/owning-resource-not-pruned=true` works, and the labels won't be overwrite by using `istioctl tag set` again.

label prod-stable thus won't be pruned

owning-resource-not-pruned label still exists

delete operator related labels also works, but the labels will come back once using `istioctl tag set` again.

clean prod-stable labels thus won't be pruned

label is overwritten once `istioctl install` again

istio / istio

istio-operator/istioctl > 1.19 unexpectedly remove MutatingWebhookConfiguration::istio-revision-tag #47423

Is this the right place to submit this?

Bug Description

Version

Additional Information

How to repro

Install Istio 1-18-1

Upgrade to istio 1-19-3

Uninstall Istio 1-18-1

Install Istio 1.19.0

Restart istio-operator

Install Istio 1.19.0

Restart istio-operator

~annotate~ label install.operator.istio.io/owning-resource-not-pruned=true works, and the labels won't be overwrite by using istioctl tag set again.

label prod-stable thus won't be pruned

owning-resource-not-pruned label still exists

delete operator related labels also works, but the labels will come back once using istioctl tag set again.

clean prod-stable labels thus won't be pruned

label is overwritten once istioctl install again

~annotate~ label `install.operator.istio.io/owning-resource-not-pruned=true` works, and the labels won't be overwrite by using `istioctl tag set` again.

delete operator related labels also works, but the labels will come back once using `istioctl tag set` again.

label is overwritten once `istioctl install` again