portworx / terraform-ibm-portworx-enterprise

Terraform Module to manage Portworx Enterprise on IKS in IBM Cloud

Apache License 2.0

1 stars 5 forks source link

Fix: Upgrade Strategy for PX-Enterprise Upgrade #41

Closed sudas-px closed 2 years ago

sudas-px commented 2 years ago

Changes

What this PR does / why we need it:

The upgrade strategy uses helm upgrade with --timeout=5h0s parameters.

Issues

If the upgrade of any one node fails, the upgrade will fail after 5 hours

Solution

Trigger the upgrade without --wait and --timeout , but monitor the daemonset status for up-to-date pods.
Configure dynamic timeout strategy which shall fail the upgrade if a single node takes more 10 mins (Benchmarked on a 6 node ROKS/Vanilla IKS) to upgrade.