Test upgrade path from istio-operators 1.17/stable -> latest/edge with Istio CNI plugin enabled

DnPlas commented 11 months ago

What needs to get done

Test the upgrade path from an Istio Control Plane without the Istio CNI Plugin to a control plane with the plugin installed, which effectively means to run juju refresh on istio-operators 1.17/stable -> latest/edge with the Istio CNI plugin enabled.

This task depends on #351.

DOD:

Provide feedback in this issue about the upgrade path, limitations and workarounds (if any)

Why it needs to get done

We must ensure our users have a clear and seamless upgrade path from our previous stable versions of the istio-operators to the newer ones once we enable the Istio CNI plugin.

syncronize-issues-to-jira[bot] commented 11 months ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5052.

This message was autogenerated

DnPlas commented 10 months ago

The upgrade logic will be added in #365, which will help with the tests in this task. Once that implementation is approved, I will provide the steps I followed to verify the upgrade actually works and is safe.

DnPlas commented 10 months ago

The following methodology was followed for testing the upgrade path between istio-operators 1.17/stable to the latest version that installs the Istio CNI plugin. After running all of them, the upgrade process completed successfully. It has been proven that these steps also work for an upgrade from the latest/edge version of the istio-operators.

NOTE: The steps to test the upgrade are based on the changes introduced in #365.

Set env variables
```
DEFAULT_GATEWAY="test-gateway"
```
Deploy istio-pilot from Charmhub juju deploy istio-pilot --channel 1.17/stable --trust
Deploy istio-gateway from Charmhub juju deploy istio-gateway istio-ingressgateway --config kind="ingress" --channel 1.17/stable --trust
Relate istio apps juju relate istio-pilot istio-ingressgateway
Wait for them to be active and idle
Build and refresh istio-pilot from this branch juju refresh istio-pilot --path=./istio-pilot_ubuntu-20.04-amd64.charm
The upgrade by itself won't install the cni-plugin, we must provide the cni configurations for the plugin to be enabled after the refresh command:

juju config istio-pilot cni-bin-dir=<path to cni bin dir> 
juju config istio-pilot cni-conf-dir=<path to cni bin dir>`

For microk8s these values are:

/var/snap/microk8s/current/opt/cni/bin
/var/snap/microk8s/current/args/cni-network

Verify the correct upgrade and installation

Both istio-operator charms go to active and idle with no error

Model        Controller  Cloud/Region        Version  SLA          Timestamp
update-test  uk8s        microk8s/localhost  3.1.7    unsupported  19:08:19+01:00

App                   Version  Status  Scale  Charm          Channel      Rev  Address         Exposed  Message
istio-ingressgateway           active      1  istio-gateway  1.17/stable  723  10.152.183.230  no       
istio-pilot                    active      1  istio-pilot                   0  10.152.183.32   no       

Unit                     Workload  Agent  Address     Ports  Message
istio-ingressgateway/0*  active    idle   10.1.91.12         
istio-pilot/0*           active    idle   10.1.91.15

An istio-cni-node-xxx Pod must be present alongside the istiod-xxx-xxx in the same namespace where the control plane was deployed originally

update-test       istiod-79fc74b448-85xwt                          1/1     Running   0          7m4s
update-test       istio-ingressgateway-workload-6dcd975985-sxzsc   1/1     Running   0          6m51s
update-test       istio-ingressgateway-0                           1/1     Running   0          7m18s
update-test       istio-pilot-0                                    1/1     Running   0          6m24s
update-test       istio-cni-node-q887j                             1/1     Running   0          4m33s

juju debug-log should show the following message right after running the refresh command:

unit-istio-pilot-0: 19:04:08 INFO unit.istio-pilot/0.juju-log Attempting to upgrade from control plane version 1.17.3 to client version 1.17.3
unit-istio-pilot-0: 19:04:08 INFO unit.istio-pilot/0.juju-log Executing `istioctl precheck`
unit-istio-pilot-0: 19:04:08 WARNING unit.istio-pilot/0.config-changed ✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
unit-istio-pilot-0: 19:04:08 WARNING unit.istio-pilot/0.config-changed   To get started, check out https://istio.io/latest/docs/setup/getting-started/
unit-istio-pilot-0: 19:04:08 INFO unit.istio-pilot/0.juju-log Executing `istioctl upgrade` for our configuration
unit-istio-pilot-0: 19:04:08 WARNING unit.istio-pilot/0.config-changed 
unit-istio-pilot-0: 19:04:09 WARNING unit.istio-pilot/0.config-changed - Processing resources for Istio core.
unit-istio-pilot-0: 19:04:09 WARNING unit.istio-pilot/0.config-changed ✔ Istio core installed
unit-istio-pilot-0: 19:04:09 WARNING unit.istio-pilot/0.config-changed - Processing resources for Istiod.
unit-istio-pilot-0: 19:04:09 WARNING unit.istio-pilot/0.config-changed ✔ Istiod installed
unit-istio-pilot-0: 19:04:09 WARNING unit.istio-pilot/0.config-changed - Processing resources for CNI.
unit-istio-pilot-0: 19:04:19 WARNING unit.istio-pilot/0.config-changed - Processing resources for CNI. Waiting for DaemonSet/update-test/istio-cni-node
unit-istio-pilot-0: 19:04:19 WARNING unit.istio-pilot/0.config-changed ✔ CNI installed
unit-istio-pilot-0: 19:04:20 WARNING unit.istio-pilot/0.config-changed - Pruning removed resources
unit-istio-pilot-0: 19:04:20 WARNING unit.istio-pilot/0.config-changed ✔ Installation completeMaking this installation the default for injection and validation.
unit-istio-pilot-0: 19:04:20 INFO unit.istio-pilot/0.juju-log Waiting for Istio upgrade to roll out in cluster
unit-istio-pilot-0: 19:04:20 INFO unit.istio-pilot/0.juju-log Found control plane version (1.17.3) matching client version - upgrade rollout complete
unit-istio-pilot-0: 19:04:20 INFO unit.istio-pilot/0.juju-log Upgrade complete.

The upgrade process is entirely handled by the istioctl binary running inside the charm container. Please refer to In-place upgrades for more information.

DnPlas commented 10 months ago

Closing this issue as the changes in #351 work for the upgrade story.

canonical / istio-operators