[BUG] A pending fail pod from different hub is generated after upgrade rancher

[albinsun]

Describe the bug Based on harvester v1.3.0, by performing a Rancher upgrade form v2.7.11 to v2.8.3-rc4, we found that a new generated pod is stuck in pending fail.

Note that new one comes from a different image rancher/harvester-cloud-provider:v0.2.0 compare to the running one registry.rancher.com/rancher/harvester-cloud-provider:v0.2.0

To Reproduce

1. Setup RKE2 cluster based on Harvester v1.3.0 + Rancher v2.7.11

   • :green_circle: Create RKE2 cluster

     
   • :green_circle: Check deployment with pvc

     
   • :green_circle: Check LB

     

2. Upgrade Rancher to v2.8.3-rc4

   • :green_circle: Cluster is Active

     
   • :red_circle: Apps are in health

     ${\color{red} \text{0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..}}$ 
   • :green_circle: Check deployment with pvc

     
   • :green_circle: Check LB

     

Expected behavior harvester-cloud-provider should be in health after upgrade

Support bundle

1. Harvester: supportbundle_cloudproviderhub_2024-03-18T06-38-10Z.zip
2. Rancher: rancher-2024-03-18_06_41_29.tar.gz

Environment

• Harvester
  • Version: v1.3.0
  • Profile: QEMU/KVM, 3 nodes (8C/16G/500G)
  • ui-source: Auto
• Rancher
  • Version: v2.7.11 -> v2.8.3-rc4
  • Profile: Helm (1C/4G)

[albinsun]

FYI, hit similiar issue during v1.2.1 but with opposite image source

• Old pod (Running): rancher/harvester-cloud-provider:v0.2.0
• New pod (Pending): registry.rancher.com/rancher/harvester-cloud-provider:v0.2.0

Context (Rancher Community -> Prime)

1. harvester-v1.2.1 + rancher-v2.7.10
2. Upgrade Rancher to v2.7.11 (Prime)

[albinsun]

Workaround

[!NOTE] Delete the old pod and let the rolling update continue.

[bk201]

cc @starbops, we need to backport https://github.com/harvester/charts/pull/239 to the release branch to fix this problem.

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• [ ] If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted? The HEP PR is at:

• [ ] Where is the reproduce steps/test steps documented? The reproduce steps/test steps are at:

• [ ] Is there a workaround for the issue? If so, where is it documented? The workaround is at:

• [x] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)? The PR is at:

  • [x] Does the PR include the explanation for the fix or the feature?

  • [x] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart? The PR for the YAML change is at: The PR for the chart change is at: harvester/charts#239 harvester/charts#245

• [ ] If labeled: area/ui Has the UI issue filed or ready to be merged? The UI issue/PR is at:

• [x] If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged? The documentation/KB PR is at: harvester/docs#584

• [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility? The compatibility issue is filed at:

[albinsun]

Test basically fine using Rancher marketplace to update harvester-cloud-provider.

Per discussed with @starbops, there are mainly 2 paths for harvester-cloud-provider chart

1. Rancher Marketplace
2. RKE2 self-contained

This ticket fix 1) but user may still hit via 2), no sure if we can inform RKE2 to use latest harvester-cloud-provider when they release or some other cross-team integrations?

cc. @bk201 and @FrankYang0529

Environment

• harvester-v1.4.0-rc2
  • Profile: Single node QEMU/KVM (amd64)
  • ui-source: Auto
• rancher-v2.8.8 => rancher-v2.9.3-alpha6
  • Profile: Helm (v1.30.5+k3s1) on QEMU/KVM (amd64)
  • cert-manager-v1.5.1
• Guest Cluster
  • Profile: Single node v1.27.16+rke2r2
  • Image: ubuntu-22.04-server-cloudimg-amd64.img

Test Case

:green_circle: Upgrade harvester-cloud-provider via Marketplace

1. Import Harvester to rancher-v2.8.8

   
2. Create v1.28.13+rke2r1 cluster

   
3. Check harvester-cloud-provider app version less than v0.2.5

   * `harvester-cloud-provider-v0.2.4` 
4. Upgrade harvester-cloud-provider to v0.2.5+

   * Upgrade to `103.0.3-up0.2.6`  
5. Should NOT have pending fail harvester-cloud-provider pod

   

:yellow_circle: harvester-cloud-provider may rollout when upgrade Rancher

1. Import Harvester to rancher-v2.8.8

   
2. Create v1.27.16 +rke2r2 cluster

   
3. Check harvester-cloud-provider version less than v0.2.5

   
4. Deploy Nginx w/ PVC

   
5. Upgrade Rancher to v2.9.3-alpha5

   
6. Check harvester-cloud-provider is v0.2.5+

   * :warning: `rancher-v2.9.3-alpha5` still use 0.2.4 by default.  * Manually upgrade to v0.2.6  

[starbops]

I noticed that the harvester-cloud-provider's image changed from registry.rancher.com/rancher/harvester-cloud-provider:v0.2.1 to rancher/harvester-cloud-provider:v0.2.1. That's the reason why the deployment was rolled out during the Rancher upgrade, but I'm unsure what caused the difference.