Closed sonnyhcl closed 11 months ago
@sonnyhcl Have you set tag after installing 1-19-3?
@hanxiaop sorry for the missing. We did re-tag 1-19-3 in our prod environment, but missed in above re-pro context. Actually the deletion happened inside istio-opereator as log shows. I've update the re-pro steps and logs
Found a simpler way to re-pro. Tried 1.19.0/1.19.1/1.19.3. It seems at least istio-operator
>= 1.19.0 will always clean MutatingWebhookConfiguration::istio-revision-tag
, which breaks all canary-upgrade scenarios.
$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.0 sh -
$ istioctl version
no ready Istio pods in "istio-system"
client version: 1.19.0
$ istioctl operator init --revision 1-19-0 --tag "1.19.0-distroless"
Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless
Operator controller will watch namespaces: istio-system
✔ Istio operator installed
✔ Installation complete
$ cat istio-operator-1.19.0.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istiocontrolplane
spec:
profile: minimal
tag: 1.19.0-distroless
revision: 1-19-0
components:
pilot:
enabled: true
ingressGateways:
- enabled: true
name: istio-ingressgateway
$ k apply -f istio-operator-1.19.0.yaml
istiooperator.install.istio.io/istiocontrolplane configured
$ istioctl tag set prod-stable --revision 1-19-0 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-0". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ istioctl tag list
TAG REVISION NAMESPACES
prod-canary 1-19-3
$ k rollout restart -n istio-system istio-operator-1-19-0
$ istioctl tag list
TAG REVISION NAMESPACES
prod-canary 1-19-3
wait 1min to let istio-operator reconciling
2023-10-18T06:36:28.336109Z info ControlZ available at 127.0.0.1:9876
2023-10-18T06:36:28.336216Z info Leader election cm: istio-operator-lock-1-19-0
2023-10-18T06:36:28.336564Z info Creating operator metrics exporter available at 127.0.0.1:15014
2023-10-18T06:36:28.336962Z info Registering Components.
2023-10-18T06:36:28.337615Z info installer Adding controller for IstioOperator.
2023-10-18T06:36:28.363785Z info installer Controller added
2023-10-18T06:36:28.363814Z info Starting the Cmd.
2023-10-18T06:36:28.364966Z info klog attempting to acquire leader lease istio-operator/istio-operator-lock-1-19-0...
2023-10-18T06:36:51.172669Z info klog successfully acquired lease istio-operator/istio-operator-lock-1-19-0
2023-10-18T06:36:52.037423Z info installer Reconciling IstioOperator
2023-10-18T06:36:52.127816Z info installer Updating IstioOperator
2023-10-18T06:36:52.128074Z info installer Detecting third-party JWT support
2023-10-18T06:36:53.745097Z info cluster "" kube client started
2023-10-18T06:36:53.845793Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T06:36:53.845882Z info kube Pilot K8S CRD controller synced in 96.702µs controller=analysis-controller
2023-10-18T06:36:53.845919Z info kube Starting Pilot K8S CRD controller controller=analysis-controller
2023-10-18T06:36:53.845933Z info kube Pilot K8S CRD controller synced in 16.4µs controller=analysis-controller
2023-10-18T06:36:53.848672Z info installer Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443
2023-10-18T06:36:53.848689Z info installer Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443
2023-10-18T06:36:53.848847Z info installer Generated manifest objects are the same as cached for component Ztunnel.
2023-10-18T06:36:53.848946Z info installer Pilot is waiting on dependency...
2023-10-18T06:36:53.848963Z info installer Cni is waiting on dependency...
2023-10-18T06:36:53.848974Z info installer Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443
2023-10-18T06:36:53.849110Z info installer Generated manifest objects are the same as cached for component IstiodRemote.
2023-10-18T06:36:53.849130Z info installer IngressGateways is waiting on dependency...
2023-10-18T06:36:53.849137Z info installer EgressGateways is waiting on dependency...
2023-10-18T06:36:54.533033Z info installer The following objects differ between generated manifest and cache:
- ServiceAccount:istio-system:istio-reader-service-account
- CustomResourceDefinition::wasmplugins.extensions.istio.io
- CustomResourceDefinition::destinationrules.networking.istio.io
- CustomResourceDefinition::envoyfilters.networking.istio.io
- CustomResourceDefinition::gateways.networking.istio.io
- CustomResourceDefinition::proxyconfigs.networking.istio.io
- CustomResourceDefinition::serviceentries.networking.istio.io
- CustomResourceDefinition::sidecars.networking.istio.io
- CustomResourceDefinition::virtualservices.networking.istio.io
- CustomResourceDefinition::workloadentries.networking.istio.io
- CustomResourceDefinition::workloadgroups.networking.istio.io
- CustomResourceDefinition::authorizationpolicies.security.istio.io
- CustomResourceDefinition::peerauthentications.security.istio.io
- CustomResourceDefinition::requestauthentications.security.istio.io
- CustomResourceDefinition::telemetries.telemetry.istio.io
- CustomResourceDefinition::istiooperators.install.istio.io
2023-10-18T06:36:54.533164Z info installer using server side apply to update obj: CustomResourceDefinition//authorizationpolicies.security.istio.io
2023-10-18T06:36:54.555213Z info installer using server side apply to update obj: CustomResourceDefinition//destinationrules.networking.istio.io
2023-10-18T06:36:54.742451Z info installer using server side apply to update obj: CustomResourceDefinition//envoyfilters.networking.istio.io
2023-10-18T06:36:54.762292Z info installer using server side apply to update obj: CustomResourceDefinition//gateways.networking.istio.io
2023-10-18T06:36:54.781690Z info installer using server side apply to update obj: CustomResourceDefinition//istiooperators.install.istio.io
2023-10-18T06:36:54.791087Z info installer using server side apply to update obj: CustomResourceDefinition//peerauthentications.security.istio.io
2023-10-18T06:36:54.805337Z info installer using server side apply to update obj: CustomResourceDefinition//proxyconfigs.networking.istio.io
2023-10-18T06:36:54.815990Z info installer using server side apply to update obj: CustomResourceDefinition//requestauthentications.security.istio.io
2023-10-18T06:36:54.837951Z info installer using server side apply to update obj: CustomResourceDefinition//serviceentries.networking.istio.io
2023-10-18T06:36:54.859512Z info installer using server side apply to update obj: CustomResourceDefinition//sidecars.networking.istio.io
2023-10-18T06:36:54.882910Z info installer using server side apply to update obj: CustomResourceDefinition//telemetries.telemetry.istio.io
2023-10-18T06:36:54.901104Z info installer using server side apply to update obj: CustomResourceDefinition//virtualservices.networking.istio.io
2023-10-18T06:36:54.994491Z info installer using server side apply to update obj: CustomResourceDefinition//wasmplugins.extensions.istio.io
2023-10-18T06:36:55.007659Z info installer using server side apply to update obj: CustomResourceDefinition//workloadentries.networking.istio.io
2023-10-18T06:36:55.019101Z info installer using server side apply to update obj: CustomResourceDefinition//workloadgroups.networking.istio.io
2023-10-18T06:36:55.042370Z info installer using server side apply to update obj: ServiceAccount/istio-system/istio-reader-service-account
2023-10-18T06:36:55.557427Z info installer established CRD authorizationpolicies.security.istio.io
2023-10-18T06:36:55.570827Z info installer established CRD destinationrules.networking.istio.io
2023-10-18T06:36:55.577036Z info installer established CRD envoyfilters.networking.istio.io
2023-10-18T06:36:55.584606Z info installer established CRD gateways.networking.istio.io
2023-10-18T06:36:55.589211Z info installer established CRD istiooperators.install.istio.io
2023-10-18T06:36:55.594265Z info installer established CRD peerauthentications.security.istio.io
2023-10-18T06:36:55.599570Z info installer established CRD proxyconfigs.networking.istio.io
2023-10-18T06:36:55.605172Z info installer established CRD requestauthentications.security.istio.io
2023-10-18T06:36:55.610211Z info installer established CRD serviceentries.networking.istio.io
2023-10-18T06:36:55.615857Z info installer established CRD sidecars.networking.istio.io
2023-10-18T06:36:55.620660Z info installer established CRD telemetries.telemetry.istio.io
2023-10-18T06:36:55.628498Z info installer established CRD virtualservices.networking.istio.io
2023-10-18T06:36:55.634131Z info installer established CRD wasmplugins.extensions.istio.io
2023-10-18T06:36:55.638900Z info installer established CRD workloadentries.networking.istio.io
2023-10-18T06:36:55.643886Z info installer established CRD workloadgroups.networking.istio.io
2023-10-18T06:36:55.643907Z info installer Finished applying CRDs.
2023-10-18T06:36:55.644125Z info installer Unblocking dependency Pilot.
2023-10-18T06:36:55.644145Z info installer Dependency for Pilot has completed, proceeding.
2023-10-18T06:36:55.644155Z info installer Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443
- Processing resources for Istio core.
2023-10-18T06:36:55.738768Z info installer The following objects differ between generated manifest and cache:
- HorizontalPodAutoscaler:istio-system:istiod-1-19-0
- ClusterRole::istiod-clusterrole-1-19-0-istio-system
- ClusterRole::istiod-gateway-controller-1-19-0-istio-system
- ClusterRoleBinding::istiod-clusterrole-1-19-0-istio-system
- ClusterRoleBinding::istiod-gateway-controller-1-19-0-istio-system
- ConfigMap:istio-system:istio-1-19-0
- Deployment:istio-system:istiod-1-19-0
- ConfigMap:istio-system:istio-sidecar-injector-1-19-0
- MutatingWebhookConfiguration::istio-sidecar-injector-1-19-0
- PodDisruptionBudget:istio-system:istiod-1-19-0
- ClusterRole::istio-reader-clusterrole-1-19-0-istio-system
- ClusterRoleBinding::istio-reader-clusterrole-1-19-0-istio-system
- Role:istio-system:istiod-1-19-0
- RoleBinding:istio-system:istiod-1-19-0
- Service:istio-system:istiod-1-19-0
- ServiceAccount:istio-system:istiod-1-19-0
- ValidatingWebhookConfiguration::istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.738846Z info installer using server side apply to update obj: ServiceAccount/istio-system/istiod-1-19-0
2023-10-18T06:36:55.834335Z info installer using server side apply to update obj: ClusterRole//istio-reader-clusterrole-1-19-0-istio-system
✔ Istio core installed
2023-10-18T06:36:55.845991Z info installer using server side apply to update obj: ClusterRole//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.854131Z info installer using server side apply to update obj: ClusterRole//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.860114Z info installer using server side apply to update obj: ClusterRoleBinding//istio-reader-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.866759Z info installer using server side apply to update obj: ClusterRoleBinding//istiod-clusterrole-1-19-0-istio-system
2023-10-18T06:36:55.873611Z info installer using server side apply to update obj: ClusterRoleBinding//istiod-gateway-controller-1-19-0-istio-system
2023-10-18T06:36:55.880284Z info installer using server side apply to update obj: ValidatingWebhookConfiguration//istio-validator-1-19-0-istio-system
2023-10-18T06:36:55.889650Z info installer using server side apply to update obj: ConfigMap/istio-system/istio-1-19-0
2023-10-18T06:36:55.903834Z info installer using server side apply to update obj: ConfigMap/istio-system/istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.929110Z info installer using server side apply to update obj: MutatingWebhookConfiguration//istio-sidecar-injector-1-19-0
2023-10-18T06:36:55.940177Z info installer using server side apply to update obj: Deployment/istio-system/istiod-1-19-0
2023-10-18T06:36:55.957152Z info installer using server side apply to update obj: PodDisruptionBudget/istio-system/istiod-1-19-0
2023-10-18T06:36:55.964597Z info installer using server side apply to update obj: Role/istio-system/istiod-1-19-0
2023-10-18T06:36:55.971591Z info installer using server side apply to update obj: RoleBinding/istio-system/istiod-1-19-0
2023-10-18T06:36:55.977989Z info installer using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istiod-1-19-0
2023-10-18T06:36:55.985071Z info installer using server side apply to update obj: Service/istio-system/istiod-1-19-0
- Processing resources for Istiod.
2023-10-18T06:36:56.007220Z info installer Unblocking dependency Cni.
2023-10-18T06:36:56.007317Z info installer Unblocking dependency IngressGateways.
2023-10-18T06:36:56.007324Z info installer Unblocking dependency EgressGateways.
2023-10-18T06:36:56.007332Z info installer Dependency for EgressGateways has completed, proceeding.
2023-10-18T06:36:56.007361Z info installer Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.007547Z info installer Generated manifest objects are the same as cached for component EgressGateways.
2023-10-18T06:36:56.007582Z info installer Dependency for Cni has completed, proceeding.
2023-10-18T06:36:56.007594Z info installer Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443
2023-10-18T06:36:56.007620Z info installer Generated manifest objects are the same as cached for component Cni.
2023-10-18T06:36:56.007625Z info installer Dependency for IngressGateways has completed, proceeding.
2023-10-18T06:36:56.007629Z info installer Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443
2023-10-18T06:36:56.012152Z info installer The following objects differ between generated manifest and cache:
- HorizontalPodAutoscaler:istio-system:istio-ingressgateway
- Deployment:istio-system:istio-ingressgateway
- PodDisruptionBudget:istio-system:istio-ingressgateway
- Role:istio-system:istio-ingressgateway-sds
- RoleBinding:istio-system:istio-ingressgateway-sds
- Service:istio-system:istio-ingressgateway
- ServiceAccount:istio-system:istio-ingressgateway-service-account
2023-10-18T06:36:56.012237Z info installer using server side apply to update obj: ServiceAccount/istio-system/istio-ingressgateway-service-account
✔ Istiod installed
2023-10-18T06:36:56.018353Z info installer using server side apply to update obj: Deployment/istio-system/istio-ingressgateway
2023-10-18T06:36:56.042772Z info installer using server side apply to update obj: PodDisruptionBudget/istio-system/istio-ingressgateway
2023-10-18T06:36:56.050471Z info installer using server side apply to update obj: Role/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.057116Z info installer using server side apply to update obj: RoleBinding/istio-system/istio-ingressgateway-sds
2023-10-18T06:36:56.064533Z info installer using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istio-ingressgateway
2023-10-18T06:36:56.072139Z info installer using server side apply to update obj: Service/istio-system/istio-ingressgateway
- Processing resources for Ingress gateways.
✔ Ingress gateways installed
2023-10-18T06:37:11.735610Z info installer Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache.
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736122Z info Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-18T06:37:11.736279Z info installer Watching a change for istio resource: /istio-revision-tag-prod-canary
Now istio revision tag has been cleaned.
$ istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list
Found a simpler way to re-pro. Tried 1.19.0/1.19.1/1.19.3. It seems at least
istio-operator
>= 1.19.0 will always cleanMutatingWebhookConfiguration::istio-revision-tag
, which breaks all canary-upgrade scenarios.Install Istio 1.19.0
$ curl -L -k https://istio.io/downloadIstioctl | ISTIO_VERSION=1.19.0 sh - $ istioctl version no ready Istio pods in "istio-system" client version: 1.19.0 $ istioctl operator init --revision 1-19-0 --tag "1.19.0-distroless" Installing operator controller in namespace: istio-operator using image: docker.io/istio/operator:1.19.3-distroless Operator controller will watch namespaces: istio-system ✔ Istio operator installed ✔ Installation complete $ cat istio-operator-1.19.0.yaml apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: namespace: istio-system name: istiocontrolplane spec: profile: minimal tag: 1.19.0-distroless revision: 1-19-0 components: pilot: enabled: true ingressGateways: - enabled: true name: istio-ingressgateway $ k apply -f istio-operator-1.19.0.yaml istiooperator.install.istio.io/istiocontrolplane configured $ istioctl tag set prod-stable --revision 1-19-0 --overwrite Revision tag "prod-stable" created, referencing control plane revision "1-19-0". To enable injection using this revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable' $ istioctl tag list TAG REVISION NAMESPACES prod-canary 1-19-3
Restart istio-operator
$ k rollout restart -n istio-system istio-operator-1-19-0 $ istioctl tag list TAG REVISION NAMESPACES prod-canary 1-19-3
wait 1min to let istio-operator reconciling
2023-10-18T06:36:28.336109Z info ControlZ available at 127.0.0.1:9876 2023-10-18T06:36:28.336216Z info Leader election cm: istio-operator-lock-1-19-0 2023-10-18T06:36:28.336564Z info Creating operator metrics exporter available at 127.0.0.1:15014 2023-10-18T06:36:28.336962Z info Registering Components. 2023-10-18T06:36:28.337615Z info installer Adding controller for IstioOperator. 2023-10-18T06:36:28.363785Z info installer Controller added 2023-10-18T06:36:28.363814Z info Starting the Cmd. 2023-10-18T06:36:28.364966Z info klog attempting to acquire leader lease istio-operator/istio-operator-lock-1-19-0... 2023-10-18T06:36:51.172669Z info klog successfully acquired lease istio-operator/istio-operator-lock-1-19-0 2023-10-18T06:36:52.037423Z info installer Reconciling IstioOperator 2023-10-18T06:36:52.127816Z info installer Updating IstioOperator 2023-10-18T06:36:52.128074Z info installer Detecting third-party JWT support 2023-10-18T06:36:53.745097Z info cluster "" kube client started 2023-10-18T06:36:53.845793Z info kube Starting Pilot K8S CRD controller controller=analysis-controller 2023-10-18T06:36:53.845882Z info kube Pilot K8S CRD controller synced in 96.702µs controller=analysis-controller 2023-10-18T06:36:53.845919Z info kube Starting Pilot K8S CRD controller controller=analysis-controller 2023-10-18T06:36:53.845933Z info kube Pilot K8S CRD controller synced in 16.4µs controller=analysis-controller 2023-10-18T06:36:53.848672Z info installer Processing resources from manifest: Ztunnel for CR istiocontrolplane-istio-system-Ztunnel-https://10.0.0.1:443 2023-10-18T06:36:53.848689Z info installer Processing resources from manifest: Base for CR istiocontrolplane-istio-system-Base-https://10.0.0.1:443 2023-10-18T06:36:53.848847Z info installer Generated manifest objects are the same as cached for component Ztunnel. 2023-10-18T06:36:53.848946Z info installer Pilot is waiting on dependency... 2023-10-18T06:36:53.848963Z info installer Cni is waiting on dependency... 2023-10-18T06:36:53.848974Z info installer Processing resources from manifest: IstiodRemote for CR istiocontrolplane-istio-system-IstiodRemote-https://10.0.0.1:443 2023-10-18T06:36:53.849110Z info installer Generated manifest objects are the same as cached for component IstiodRemote. 2023-10-18T06:36:53.849130Z info installer IngressGateways is waiting on dependency... 2023-10-18T06:36:53.849137Z info installer EgressGateways is waiting on dependency... 2023-10-18T06:36:54.533033Z info installer The following objects differ between generated manifest and cache: - ServiceAccount:istio-system:istio-reader-service-account - CustomResourceDefinition::wasmplugins.extensions.istio.io - CustomResourceDefinition::destinationrules.networking.istio.io - CustomResourceDefinition::envoyfilters.networking.istio.io - CustomResourceDefinition::gateways.networking.istio.io - CustomResourceDefinition::proxyconfigs.networking.istio.io - CustomResourceDefinition::serviceentries.networking.istio.io - CustomResourceDefinition::sidecars.networking.istio.io - CustomResourceDefinition::virtualservices.networking.istio.io - CustomResourceDefinition::workloadentries.networking.istio.io - CustomResourceDefinition::workloadgroups.networking.istio.io - CustomResourceDefinition::authorizationpolicies.security.istio.io - CustomResourceDefinition::peerauthentications.security.istio.io - CustomResourceDefinition::requestauthentications.security.istio.io - CustomResourceDefinition::telemetries.telemetry.istio.io - CustomResourceDefinition::istiooperators.install.istio.io 2023-10-18T06:36:54.533164Z info installer using server side apply to update obj: CustomResourceDefinition//authorizationpolicies.security.istio.io 2023-10-18T06:36:54.555213Z info installer using server side apply to update obj: CustomResourceDefinition//destinationrules.networking.istio.io 2023-10-18T06:36:54.742451Z info installer using server side apply to update obj: CustomResourceDefinition//envoyfilters.networking.istio.io 2023-10-18T06:36:54.762292Z info installer using server side apply to update obj: CustomResourceDefinition//gateways.networking.istio.io 2023-10-18T06:36:54.781690Z info installer using server side apply to update obj: CustomResourceDefinition//istiooperators.install.istio.io 2023-10-18T06:36:54.791087Z info installer using server side apply to update obj: CustomResourceDefinition//peerauthentications.security.istio.io 2023-10-18T06:36:54.805337Z info installer using server side apply to update obj: CustomResourceDefinition//proxyconfigs.networking.istio.io 2023-10-18T06:36:54.815990Z info installer using server side apply to update obj: CustomResourceDefinition//requestauthentications.security.istio.io 2023-10-18T06:36:54.837951Z info installer using server side apply to update obj: CustomResourceDefinition//serviceentries.networking.istio.io 2023-10-18T06:36:54.859512Z info installer using server side apply to update obj: CustomResourceDefinition//sidecars.networking.istio.io 2023-10-18T06:36:54.882910Z info installer using server side apply to update obj: CustomResourceDefinition//telemetries.telemetry.istio.io 2023-10-18T06:36:54.901104Z info installer using server side apply to update obj: CustomResourceDefinition//virtualservices.networking.istio.io 2023-10-18T06:36:54.994491Z info installer using server side apply to update obj: CustomResourceDefinition//wasmplugins.extensions.istio.io 2023-10-18T06:36:55.007659Z info installer using server side apply to update obj: CustomResourceDefinition//workloadentries.networking.istio.io 2023-10-18T06:36:55.019101Z info installer using server side apply to update obj: CustomResourceDefinition//workloadgroups.networking.istio.io 2023-10-18T06:36:55.042370Z info installer using server side apply to update obj: ServiceAccount/istio-system/istio-reader-service-account 2023-10-18T06:36:55.557427Z info installer established CRD authorizationpolicies.security.istio.io 2023-10-18T06:36:55.570827Z info installer established CRD destinationrules.networking.istio.io 2023-10-18T06:36:55.577036Z info installer established CRD envoyfilters.networking.istio.io 2023-10-18T06:36:55.584606Z info installer established CRD gateways.networking.istio.io 2023-10-18T06:36:55.589211Z info installer established CRD istiooperators.install.istio.io 2023-10-18T06:36:55.594265Z info installer established CRD peerauthentications.security.istio.io 2023-10-18T06:36:55.599570Z info installer established CRD proxyconfigs.networking.istio.io 2023-10-18T06:36:55.605172Z info installer established CRD requestauthentications.security.istio.io 2023-10-18T06:36:55.610211Z info installer established CRD serviceentries.networking.istio.io 2023-10-18T06:36:55.615857Z info installer established CRD sidecars.networking.istio.io 2023-10-18T06:36:55.620660Z info installer established CRD telemetries.telemetry.istio.io 2023-10-18T06:36:55.628498Z info installer established CRD virtualservices.networking.istio.io 2023-10-18T06:36:55.634131Z info installer established CRD wasmplugins.extensions.istio.io 2023-10-18T06:36:55.638900Z info installer established CRD workloadentries.networking.istio.io 2023-10-18T06:36:55.643886Z info installer established CRD workloadgroups.networking.istio.io 2023-10-18T06:36:55.643907Z info installer Finished applying CRDs. 2023-10-18T06:36:55.644125Z info installer Unblocking dependency Pilot. 2023-10-18T06:36:55.644145Z info installer Dependency for Pilot has completed, proceeding. 2023-10-18T06:36:55.644155Z info installer Processing resources from manifest: Pilot for CR istiocontrolplane-istio-system-Pilot-https://10.0.0.1:443 - Processing resources for Istio core. 2023-10-18T06:36:55.738768Z info installer The following objects differ between generated manifest and cache: - HorizontalPodAutoscaler:istio-system:istiod-1-19-0 - ClusterRole::istiod-clusterrole-1-19-0-istio-system - ClusterRole::istiod-gateway-controller-1-19-0-istio-system - ClusterRoleBinding::istiod-clusterrole-1-19-0-istio-system - ClusterRoleBinding::istiod-gateway-controller-1-19-0-istio-system - ConfigMap:istio-system:istio-1-19-0 - Deployment:istio-system:istiod-1-19-0 - ConfigMap:istio-system:istio-sidecar-injector-1-19-0 - MutatingWebhookConfiguration::istio-sidecar-injector-1-19-0 - PodDisruptionBudget:istio-system:istiod-1-19-0 - ClusterRole::istio-reader-clusterrole-1-19-0-istio-system - ClusterRoleBinding::istio-reader-clusterrole-1-19-0-istio-system - Role:istio-system:istiod-1-19-0 - RoleBinding:istio-system:istiod-1-19-0 - Service:istio-system:istiod-1-19-0 - ServiceAccount:istio-system:istiod-1-19-0 - ValidatingWebhookConfiguration::istio-validator-1-19-0-istio-system 2023-10-18T06:36:55.738846Z info installer using server side apply to update obj: ServiceAccount/istio-system/istiod-1-19-0 2023-10-18T06:36:55.834335Z info installer using server side apply to update obj: ClusterRole//istio-reader-clusterrole-1-19-0-istio-system ✔ Istio core installed 2023-10-18T06:36:55.845991Z info installer using server side apply to update obj: ClusterRole//istiod-clusterrole-1-19-0-istio-system 2023-10-18T06:36:55.854131Z info installer using server side apply to update obj: ClusterRole//istiod-gateway-controller-1-19-0-istio-system 2023-10-18T06:36:55.860114Z info installer using server side apply to update obj: ClusterRoleBinding//istio-reader-clusterrole-1-19-0-istio-system 2023-10-18T06:36:55.866759Z info installer using server side apply to update obj: ClusterRoleBinding//istiod-clusterrole-1-19-0-istio-system 2023-10-18T06:36:55.873611Z info installer using server side apply to update obj: ClusterRoleBinding//istiod-gateway-controller-1-19-0-istio-system 2023-10-18T06:36:55.880284Z info installer using server side apply to update obj: ValidatingWebhookConfiguration//istio-validator-1-19-0-istio-system 2023-10-18T06:36:55.889650Z info installer using server side apply to update obj: ConfigMap/istio-system/istio-1-19-0 2023-10-18T06:36:55.903834Z info installer using server side apply to update obj: ConfigMap/istio-system/istio-sidecar-injector-1-19-0 2023-10-18T06:36:55.929110Z info installer using server side apply to update obj: MutatingWebhookConfiguration//istio-sidecar-injector-1-19-0 2023-10-18T06:36:55.940177Z info installer using server side apply to update obj: Deployment/istio-system/istiod-1-19-0 2023-10-18T06:36:55.957152Z info installer using server side apply to update obj: PodDisruptionBudget/istio-system/istiod-1-19-0 2023-10-18T06:36:55.964597Z info installer using server side apply to update obj: Role/istio-system/istiod-1-19-0 2023-10-18T06:36:55.971591Z info installer using server side apply to update obj: RoleBinding/istio-system/istiod-1-19-0 2023-10-18T06:36:55.977989Z info installer using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istiod-1-19-0 2023-10-18T06:36:55.985071Z info installer using server side apply to update obj: Service/istio-system/istiod-1-19-0 - Processing resources for Istiod. 2023-10-18T06:36:56.007220Z info installer Unblocking dependency Cni. 2023-10-18T06:36:56.007317Z info installer Unblocking dependency IngressGateways. 2023-10-18T06:36:56.007324Z info installer Unblocking dependency EgressGateways. 2023-10-18T06:36:56.007332Z info installer Dependency for EgressGateways has completed, proceeding. 2023-10-18T06:36:56.007361Z info installer Processing resources from manifest: EgressGateways for CR istiocontrolplane-istio-system-EgressGateways-https://10.0.0.1:443 2023-10-18T06:36:56.007547Z info installer Generated manifest objects are the same as cached for component EgressGateways. 2023-10-18T06:36:56.007582Z info installer Dependency for Cni has completed, proceeding. 2023-10-18T06:36:56.007594Z info installer Processing resources from manifest: Cni for CR istiocontrolplane-istio-system-Cni-https://10.0.0.1:443 2023-10-18T06:36:56.007620Z info installer Generated manifest objects are the same as cached for component Cni. 2023-10-18T06:36:56.007625Z info installer Dependency for IngressGateways has completed, proceeding. 2023-10-18T06:36:56.007629Z info installer Processing resources from manifest: IngressGateways for CR istiocontrolplane-istio-system-IngressGateways-https://10.0.0.1:443 2023-10-18T06:36:56.012152Z info installer The following objects differ between generated manifest and cache: - HorizontalPodAutoscaler:istio-system:istio-ingressgateway - Deployment:istio-system:istio-ingressgateway - PodDisruptionBudget:istio-system:istio-ingressgateway - Role:istio-system:istio-ingressgateway-sds - RoleBinding:istio-system:istio-ingressgateway-sds - Service:istio-system:istio-ingressgateway - ServiceAccount:istio-system:istio-ingressgateway-service-account 2023-10-18T06:36:56.012237Z info installer using server side apply to update obj: ServiceAccount/istio-system/istio-ingressgateway-service-account ✔ Istiod installed 2023-10-18T06:36:56.018353Z info installer using server side apply to update obj: Deployment/istio-system/istio-ingressgateway 2023-10-18T06:36:56.042772Z info installer using server side apply to update obj: PodDisruptionBudget/istio-system/istio-ingressgateway 2023-10-18T06:36:56.050471Z info installer using server side apply to update obj: Role/istio-system/istio-ingressgateway-sds 2023-10-18T06:36:56.057116Z info installer using server side apply to update obj: RoleBinding/istio-system/istio-ingressgateway-sds 2023-10-18T06:36:56.064533Z info installer using server side apply to update obj: HorizontalPodAutoscaler/istio-system/istio-ingressgateway 2023-10-18T06:36:56.072139Z info installer using server side apply to update obj: Service/istio-system/istio-ingressgateway - Processing resources for Ingress gateways. ✔ Ingress gateways installed 2023-10-18T06:37:11.735610Z info installer Removed object MutatingWebhookConfiguration::istio-revision-tag-prod-canary from Cache. Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary. 2023-10-18T06:37:11.736122Z info Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary. 2023-10-18T06:37:11.736279Z info installer Watching a change for istio resource: /istio-revision-tag-prod-canary
Now istio revision tag has been cleaned.
$ istioctl tag list No Istio revision tag MutatingWebhookConfigurations to list
@sonnyhcl Thanks for the info! This seems like a regression. I'll look into it.
BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.
We observe a similar issue with istioctl install. Ways to reproduce:
istioctl tag list
TAG REVISION NAMESPACES
default 1-19 a,b,c,d,e
✔ Istio core installed
✔ Istiod installed
✔ CNI installed
✔ Ingress gateways installed
- Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-default.
Removed ValidatingWebhookConfiguration::istiod-default-validator.
✔ Installation complete
istioctl tag list
No Istio revision tag MutatingWebhookConfigurations to list
Also observed istioctl install
will prune user set revision tag.
2023-10-24 01:55:41.664295: ++ istioctl install -y --verify --revision 1-19-3 -f setup/istio-operator.yaml
2023-10-24 01:55:42.422813: WARNING: Istio is being upgraded from 1.19.0 to 1.19.3.
2023-10-24 01:55:42.423134: Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings.
2023-10-24 01:55:43.364625:
2023-10-24 01:55:45.723057: - Processing resources for Istio core.
2023-10-24 01:55:45.820562: \N{HEAVY CHECK MARK} Istio core installed
2023-10-24 01:55:47.009512: - Processing resources for Istiod.
2023-10-24 01:55:47.069855: \N{HEAVY CHECK MARK} Istiod installed
2023-10-24 01:55:47.467949: - Processing resources for Ingress gateways.
2023-10-24 01:55:47.469677: \N{HEAVY CHECK MARK} Ingress gateways installed
2023-10-24 01:55:52.979047: - Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
2023-10-24 01:55:53.026674: Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
This is blocking us upgrading to istio 1.19.3, which is a security fix. @hanxiaop do you know any other workaround? considering it will take time to release a new istio version.
Also observed
istioctl install
will prune user set revision tag.2023-10-24 01:55:41.664295: ++ istioctl install -y --verify --revision 1-19-3 -f setup/istio-operator.yaml 2023-10-24 01:55:42.422813: WARNING: Istio is being upgraded from 1.19.0 to 1.19.3. 2023-10-24 01:55:42.423134: Before upgrading, you may wish to use 'istioctl x precheck' to check for upgrade warnings. 2023-10-24 01:55:43.364625: 2023-10-24 01:55:45.723057: - Processing resources for Istio core. 2023-10-24 01:55:45.820562: \N{HEAVY CHECK MARK} Istio core installed 2023-10-24 01:55:47.009512: - Processing resources for Istiod. 2023-10-24 01:55:47.069855: \N{HEAVY CHECK MARK} Istiod installed 2023-10-24 01:55:47.467949: - Processing resources for Ingress gateways. 2023-10-24 01:55:47.469677: \N{HEAVY CHECK MARK} Ingress gateways installed 2023-10-24 01:55:52.979047: - Pruning removed resources Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary. 2023-10-24 01:55:53.026674: Removed MutatingWebhookConfiguration::istio-revision-tag-prod-stable.
This is blocking us upgrading to istio 1.19.3, which is a security fix. @hanxiaop do you know any other workaround? considering it will take time to release a new istio version.
@sonnyhcl I'm going to fix it up soon. Before having the fix, maybe you can try to set the annotation install.operator.istio.io/owning-resource-not-pruned="true"
to keep the resource?
If it's not working, deleting the operator related labels (starting with install.operator.istio.io
) of the tag webhook you've created with istioctl
.
both workarounds are fine, with small diff.
install.operator.istio.io/owning-resource-not-pruned=true
works, and the labels won't be overwrite by using istioctl tag set
again.$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ istioctl tag set prod-canary --revision 1-19-3
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'
$ k get MutatingWebhookConfigurations
NAME WEBHOOKS AGE
istio-revision-tag-prod-canary 2 100s
istio-revision-tag-prod-stable 2 97s
istio-sidecar-injector-1-19-3 2 4d1h
$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-not-pruned=true --overwrite
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable labeled
$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
install.operator.istio.io/owning-resource-namespace: istio-system
install.operator.istio.io/owning-resource-not-pruned: "true"
$ istioctl install -y --revision 1-19-3 --verify -f istio-operator-1.19.3.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
- Pruning removed resources
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
✔ Installation complete
$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
install.operator.istio.io/owning-resource-namespace: istio-system
install.operator.istio.io/owning-resource-not-pruned: "true"
istioctl tag set
again.$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ istioctl tag set prod-canary --revision 1-19-3 --overwrite
Revision tag "prod-canary" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-canary'
$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-namespace-
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable unlabeled
$ kubectl label MutatingWebhookConfigurations istio-revision-tag-prod-stable install.operator.istio.io/owning-resource-
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-revision-tag-prod-stable unlabeled
$ istioctl install -y --revision 1-19-3 --verify -f istio-operator-1.19.3.yaml
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
- Pruning removed resources
Removed MutatingWebhookConfiguration::istio-revision-tag-prod-canary.
✔ Installation complete
istioctl install
again$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
install.operator.istio.io/owning-resource-not-pruned: "true"
$ istioctl tag set prod-stable --revision 1-19-3 --overwrite
Revision tag "prod-stable" created, referencing control plane revision "1-19-3". To enable injection using this
revision tag, use 'kubectl label namespace <NAMESPACE> istio.io/rev=prod-stable'
$ k get MutatingWebhookConfigurations -o yaml istio-revision-tag-prod-stable | grep install.operator
install.operator.istio.io/owning-resource: installed-state-istiocontrolplane-1-19-3
install.operator.istio.io/owning-resource-namespace: istio-system
install.operator.istio.io/owning-resource-not-pruned: "true"
@sonnyhcl Thanks for the verification, I've submitted a fix and I think it can be included in the next release.
BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.
Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?
BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.
Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?
I mean the in-cluster operator controller is not encouraged, which is deployed by istioctl operator init
. Currently we encourage people to use Helm, or istioctl to install Istio.
BTW the operator is not encouraged and is lack of maintenance for a long time; we suggest switching to other installation methods.
Thanks for this info. what do you mean? "istioctl operator init --revision 1-18-1" this is not encouraged?
I mean the in-cluster operator controller is not encouraged, which is deployed by
istioctl operator init
. Currently we encourage people to use Helm, or istioctl to install Istio.
Thanks for clarification. Now I understand the difference. But somehow myself and my team likes that in-cluster operator and we are struck with it. If we have to move from that to "helm" it is big work for us.
Anyways, I hit with a bug on istioctl binary of 1.18.5 version. If I run the below command, i get an error
istioctl tag set stable --revision 1-18-5 --overwrite
Error: failed to apply tag webhook MutatingWebhookConfiguration to cluster: failed to apply tag manifests to cluster: failed applying manifest /var/folders/yb/7gsxhlrn7t97nnrh5b565bhdscxz7p/T/revision-tag-manifest-3851079461: error validating "/var/folders/yb/7gsxhlrn7t97nnrh5b565bhdscxz7p/T/revision-tag-manifest-3851079461": error validating data: the server has asked for the client to provide credentials; if you choose to ignore these errors, turn validation off with --validate=false:
But same command with 1.19.3 binary, it is successful.
May I know is the fix (#47538) included in the 1.19.4 release train or not? @hanxiaop
May I know is the fix (#47538) included in the 1.19.4 release train or not? @hanxiaop
Looks like it's not. Will be included in the next release.
Hi @hanxiaop it looks like it was released with 1.19.5.
Unfortunately, we still see this issue. I described in https://github.com/istio/istio/issues/47423#issuecomment-1774482179 how to reproduce.
Hi @hanxiaop it looks like it was released with 1.19.5.
Unfortunately, we still see this issue. I described in #47423 (comment) how to reproduce.
@wiegandf Sent the wrong fix, which was not actually working, and the release note was included. I've already fixed this recently, and the fix will be available in 1.19.7 and 1.20.3.
If it's not working, deleting the operator related labels (starting with install.operator.istio.io) of the tag webhook you've created with istioctl.
@wiegandf This can be a workaround before waiting for the new release.
Is this the right place to submit this?
Bug Description
We are following Istio Canary Upgrades and found istioctl uninstall will unexpected remove the revision-tag created for canary upgrades.
Version
Additional Information
How to repro
Install Istio 1-18-1
Upgrade to istio 1-19-3
Uninstall Istio 1-18-1
After above command run, istio-operator log shows istio-operator deleted the revision-tag