[BUG] Seeder add-on does not get upgraded during Harvester upgrade v1.2.1 to v1.2.2

[starbops]

Describe the bug

When upgrading Harvester clusters from v1.2.1 to v1.2.2, the harvester-seeder Addon is not upgraded. This could be observed by checking the harvester-seeder Addon manifest:

$ kubectl -n harvester-system get addons.harvesterhci harvester-seeder -o yaml
apiVersion: harvesterhci.io/v1beta1
kind: Addon
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"harvesterhci.io/v1beta1","kind":"Addon","labels":{"addon.harvesterhci.io/experimental":"true"},"metadata":{"annotations":{},"name":"harvester-seeder","namespace":"harvester-system"},"spec":{"chart":"harvester-seeder","enabled":false,"repo":"http://harvester-cluster-repo.cattle-system.svc/charts","valuesContent":"image:\n  tag: v0.1.0\nfullnameOverride: harvester-seeder\n","version":"0.1.0"}}
  creationTimestamp: "2024-05-21T02:25:17Z"
  generation: 5
  name: harvester-seeder
  namespace: harvester-system
  resourceVersion: "110434"
  uid: 32a61f88-e6c5-4dbd-8398-1dfdbf8af336
spec:
  chart: harvester-seeder
  enabled: true
  repo: http://harvester-cluster-repo.cattle-system.svc/charts
  valuesContent: |
    image:
      tag: v0.1.2
    fullnameOverride: harvester-seeder
  version: 0.1.0
status:
  conditions:
  - lastUpdateTime: "2024-05-21T03:01:16Z"
    status: "False"
    type: Completed
  - lastUpdateTime: "2024-05-21T03:10:43Z"
    status: "False"
    type: InProgress
  - lastUpdateTime: "2024-05-21T03:10:43Z"
    message: addon deployment job helm-install-harvester-seeder failed
    reason: Error
    status: "True"
    type: OperationFailed
  status: AddonUpdating

The image tag was updated as per #5609, but the chart version wasn't. The 0.1.0 chart of harvester-seeder is no longer packaged into the rancher/harvester-cluster-repo:v1.2.2 image, it's 0.1.1 instead:

harvester-cluster-repo-5d9549d794-n2r46:/ # ls /srv/www/htdocs/charts/
harvester-1.2.2.tgz           harvester-seeder-0.1.1.tgz            rancher-logging-103.0.0+up3.17.10.tgz   rancher-monitoring-crd-103.0.3+up45.31.1.tgz
harvester-crd-1.2.2.tgz           harvester-vm-import-controller-0.1.8.tgz  rancher-logging-crd-103.0.0+up3.17.10.tgz
harvester-pcidevices-controller-0.2.7.tgz  index.yaml                   rancher-monitoring-103.0.3+up45.31.1.tgz

If users have the harvester-seeder Addon enabled, during the Harvester upgrade, they'll see a Helm Pod trying to upgrade the harvester-seeder chart but failing with a chart-not-found error:

$ kubectl -n harvester-system logs helm-install-harvester-seeder-6l8ch
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
    echo "KUBERNETES_SERVICE_HOST is using IPv6"
    CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
    CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' != \t\r\u\e ]]
+ export + HELM_HOST=127.0.0.1:44134
tiller --listen=127.0.0.1:44134 --storage=secret
+ HELM_HOST=127.0.0.1:44134
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
$HELM_HOME has been configured at /home/klipper-helm/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ timeout -s KILL 30 helm_v2 ls --all '^harvester-seeder$' --output json
++ jq -r '.Releases | length'
[main] 2024/05/21 04:55:53 Starting Tiller v2.17.0 (tls=false)
[main] 2024/05/21 04:55:53 GRPC listening on 127.0.0.1:44134
[main] 2024/05/21 04:55:53 Probes listening on :44135
[main] 2024/05/21 04:55:53 Storage driver is Secret
[main] 2024/05/21 04:55:53 Max history per release is 0
[storage] 2024/05/21 04:55:53 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/harvester-seeder.tgz.base64
+ CHART_PATH=/tmp/harvester-seeder.tgz
+ [[ ! -f /chart/harvester-seeder.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ [[ harvester-seeder/harvester-seeder == stable/* ]]
+ [[ -n http://harvester-cluster-repo.cattle-system.svc/charts ]]
+ [[ -f /auth/username ]]
+ helm_v3 repo add harvester-seeder http://harvester-cluster-repo.cattle-system.svc/charts
"harvester-seeder" already exists with the same configuration, skipping
+ helm_v3 repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "harvester-seeder" chart repository
Update Complete. ⎈Happy Helming!⎈
+ helm_update install --version 0.1.0
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^harvester-seeder$' --namespace harvester-system --output json
++ jq -r '"\(.[0].chart),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=harvester-seeder-0.1.0,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ harvester-seeder-0.1.0 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
+ [[ deployed == \d\e\p\l\o\y\e\d ]]
+ echo 'Already installed harvester-seeder'
Already installed harvester-seeder
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis harvester-seeder --namespace harvester-system
2024/05/21 04:55:53 Release 'harvester-seeder' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2024/05/21 04:55:53 Get release 'harvester-seeder' latest version.
2024/05/21 04:55:53 Check release 'harvester-seeder' for deprecated or removed APIs...
2024/05/21 04:55:53 Finished checking release 'harvester-seeder' for deprecated or removed APIs.
2024/05/21 04:55:53 Release 'harvester-seeder' has no deprecated or removed APIs.
2024/05/21 04:55:53 Map of release 'harvester-seeder' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading harvester-seeder'
+ shift 1
+ helm_v3 upgrade --version 0.1.0 harvester-seeder harvester-seeder/harvester-seeder --values /config/values-01_HelmChart.yaml
Upgrading harvester-seeder
Error: chart "harvester-seeder" matching 0.1.0 not found in harvester-seeder index. (try 'helm repo update'): no chart version found for harvester-seeder-0.1.0

It's worth noting that this does not jeopardize the entire Harvester upgrade. In my case, the upgrade still ended with success; it's just that the harvester-seeder add-on failed to upgrade.

To Reproduce Steps to reproduce the behavior:

1. Prepare a Harvester cluster in v1.2.1
2. Enable the harvester-seeder Addon
3. Initiate an upgrade to v1.2.2
4. Wait til the upgrade ends
5. Check the harvester-seeder Pod, it's still with image tag v0.1.0
6. There should be a crashing Pod trying to upgrade the harvester-seeder chart

Expected behavior

The harvester-seeder Addon should be upgraded successfully.

Support bundle

Harvester Support Bundle May 21 2024.zip hvst-upgrade-84rsg-upgradelog-archive-2024-05-21T03-33-55Z.zip

Environment

• Harvester ISO version: v1.2.1 upgrade to v1.2.2
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): KVM

Additional context Add any other context about the problem here.

This incident was first found in #5828, but it's not the root cause of that issue.

---

The workaround is to manually update the harvester-seeder Addon manifest (if it's already enabled, please disable it before patching):

kubectl -n harvester-system patch addons.harvesterhci harvester-seeder --type=json -p '[{"op": "replace", "path": "/spec/version", "value": "0.1.1"}]'

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• [ ] If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted? The HEP PR is at:

• [ ] Where is the reproduce steps/test steps documented? The reproduce steps/test steps are at:

• [ ] Is there a workaround for the issue? If so, where is it documented? The workaround is at:

• [ ] Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)? The PR is at:

  • [ ] Does the PR include the explanation for the fix or the feature?

  • [ ] Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart? The PR for the YAML change is at: The PR for the chart change is at:

• [ ] If labeled: area/ui Has the UI issue filed or ready to be merged? The UI issue/PR is at:

• [ ] If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged? The documentation/KB PR is at:

• [ ] If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• [ ] If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility? The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#1286

[w13915984028]

@starbops Could we add a script in the build to check the matching of addons and the packed charts, this may happen from time to time. Thanks.

[starbops]

AFAIK, there are no release branches for add-ons, so it's hard to keep track of which add-on versions go with which versions of Harvester now. Do you have any suggestions?

It seems #4937 is a good place for the discussion.

[w13915984028]

@starbops I added https://github.com/harvester/harvester-installer/pull/750 to check the version matching between addon and repo.

[lanfon72]

milestone/release required.