[BUG] Upgrade v1.2.1->v1.2.2 stuck at Pre-drained node due to certificates expiration

[W0n9]

Describe the bug

To Reproduce Steps to reproduce the behavior:

1. Upgrade harvester from v1.2.1 to v1.2.2

Expected behavior

Cluster should be upgraded successfully

Support bundle

hvst-upgrade-hjw2p-upgradelog-archive-2024-05-17T07-59-36Z.zip

Environment

• Harvester ISO version: v1.2.2
• Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Baremetal with Dell PowerEdge R730

Additional context Add any other context about the problem here.

[irishgordo]

@ibrokethecloud noticing:

Already installed harvester-seeder
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis harvester-seeder --namespace harvester-system
2024/05/17 07:20:28 Release 'harvester-seeder' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2024/05/17 07:20:28 Get release 'harvester-seeder' latest version.
2024/05/17 07:20:28 Check release 'harvester-seeder' for deprecated or removed APIs...
2024/05/17 07:20:28 Finished checking release 'harvester-seeder' for deprecated or removed APIs.
2024/05/17 07:20:28 Release 'harvester-seeder' has no deprecated or removed APIs.
2024/05/17 07:20:28 Map of release 'harvester-seeder' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
+ echo 'Upgrading harvester-seeder'
+ shift 1
Upgrading harvester-seeder
+ helm_v3 upgrade --repo http://harvester-cluster-repo.cattle-system.svc/charts --version 0.1.0 harvester-seeder harvester-seeder --values /config/values-01_HelmChart.yaml
Error: chart "harvester-seeder" version "0.1.0" not found in http://harvester-cluster-repo.cattle-system.svc/charts repository

Specifically, the Error: chart "harvester-seeder" version "0.1.0" not found in http://harvester-cluster-repo.cattle-system.svc/charts repository about 6/7 times in: harvester-system.helm-install-harvester-seeder-9gccc.helm.20240517.log (extracted from .gz)

But the pre-drain on dc-0 looks to be successful:

nvim harvester-system.hvst-upgrade-hjw2p-pre-drain-harvester-dc-0-xsdr2.apply.20240517.log 

With the tail-end being:

Volume pvc-bf6c793f-7597-4dcb-9842-a9e862d05d72 is healthy.
Checking running engine pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a-e-ec25015f...
+ echo Checking running engine pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a-e-ec25015f...
+ '[' true ']'
+ '[' 4 -gt 2 ']'
++ kubectl get volumes.longhorn.io/pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a -n longhorn-system -o 'jsonpath={.status.robustness}'
+ robustness=healthy
+ '[' healthy = healthy ']'
+ echo 'Volume pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a is healthy.'
Volume pvc-fd1960b3-ef2c-4bcb-9c45-72322945648a is healthy.
+ break
+ read -r lh_engine lh_volume
+ upgrade-helper vm-live-migrate-detector harvester-dc-0 --shutdown
time="2024-05-17T07:14:02Z" level=info msg="Starting VM Live Migrate Detector"
time="2024-05-17T07:14:02Z" level=info msg="Non-migratable VM(s): []"
+ kubectl taint node harvester-dc-0 --overwrite kubevirt.io/drain=draining:NoSchedule
node/harvester-dc-0 modified
+ wait_vms_out_or_shutdown
+ local vm_count
+ '[' true ']'
++ get_running_vm_count
++ local vmioutput=/tmp/vmioutput.yaml
++ rm -f /tmp/vmioutput.yaml
++ local EXIT_CODE=0
++ kubectl get vmi -A -l kubevirt.io/nodeName=harvester-dc-0 -o yaml
++ [[ 0 -eq 0 ]]
+++ yq e '.items |  map(select(.status.phase!="Succeeded")) | length' /tmp/vmioutput.yaml
++ local count=0
++ echo 0
++ rm -f /tmp/vmioutput.yaml
+ vm_count=0
+ '[' 0 = 0 ']'
+ break
+ echo 'all VMs on node harvester-dc-0 have been live-migrated or shutdown'
+ wait_evacuation_pdb_gone
all VMs on node harvester-dc-0 have been live-migrated or shutdown
+ kubectl get pdb -o name -A
+ grep kubevirt-migration-pdb-kubevirt-evacuation-
+ patch_logging_event_audit
+ detect_upgrade
++ kubectl get upgrades.harvesterhci.io hvst-upgrade-hjw2p -n harvester-system -o yaml
+ upgrade_obj='apiVersion: harvesterhci.io/v1beta1

[Vicente-Cheng]

Hi @W0n9, Could you also generate a support bundle for investigation? Ref: https://docs.harvesterhci.io/v1.2/troubleshooting/harvester/#generate-a-support-bundle

[W0n9]

Here is my support bundle @Vicente-Cheng

Yesterday, after upgrade failed: supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z.001.zip supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z.002.zip

Today, after some fix but failed again: supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-18T02-54-16Z.001.zip supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-18T02-54-16Z.002.zip

[Vicente-Cheng]

Hi @W0n9,

Could you also provide the command to unzip this? I use 7-zip, but it looks like it is not correct.

If the SB is too large to upload, you could send the bundle to harvester-support-bundle@suse.com with this issue ID. Thanks!

[W0n9]

Hi, @Vicente-Cheng

You can try to rename it from file_name.001.zip to file_name.zip.001 .

I have sent e-mail to you.

[bk201]

@starbops Please help check this issue, thanks.

[starbops]

Hi @W0n9, I couldn't find any YAML files in the support bundle files you provided, so it's hard to determine the cluster's precise state. Did you ever change the support-bundle-namespaces setting before generating the support bundle file?

Under the Advanced -> Settings page of the dashboard:

Please also help provide the Upgrade and Setting CRs. You could run the following commands and get the output:

kubectl -n harvester-system get upgrades.harvesterhci -o yaml
kubectl get settings.harvesterhci -o yaml

From the rancher logs (supportbundle_7e8aefe3-479a-4fa4-97d8-af2f283e4577_2024-05-17T09-01-49Z/logs/cattle-system/rancher-654c45c575-zrmkv/rancher.log) we found the following line was repeated again and again:

024-05-17T08:55:00.616288694Z 2024/05/17 08:55:00 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-fb4dda4acb64: waiting for probes: kube-controller-manager, kube-scheduler

It might be related to certificates expiration. Perhaps you could check if this works for you. Thanks.

[W0n9]

Hi, @starbops I never change the support-bundle-namespaces and keep it be default. Here is the YAML:

apiVersion: v1
items:
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-09-09T06:41:47Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 41
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-f47wk
    namespace: harvester-system
    resourceVersion: "290443078"
    uid: 4382668b-352c-4263-a827-806f60e5b4c7
  spec:
    image: ""
    logEnabled: true
    version: v1.2.0
  status:
    conditions:
    - lastUpdateTime: "2023-09-09T11:52:45Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-09-09T06:42:07Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2023-09-09T06:44:00Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-09-09T06:46:24Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-09-09T07:21:42Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-09-09T10:25:23Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-09-09T11:52:45Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/harvester-iso-7x6wp
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
      harvester-dc-3:
        state: Succeeded
    previousVersion: v1.1.2
    repoInfo: |
      release:
          harvester: v1.2.0
          harvesterChart: 1.2.0
          os: Harvester v1.2.0
          kubernetes: v1.25.9+rke2r1
          rancher: v2.7.5
          monitoringChart: 102.0.0+up40.1.2
          minUpgradableVersion: v1.1.2
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2024-05-17T06:39:35Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 22
    labels:
      harvesterhci.io/latestUpgrade: "true"
      harvesterhci.io/upgradeState: UpgradingNodes
    name: hvst-upgrade-hjw2p
    namespace: harvester-system
    resourceVersion: "599965758"
    uid: 7e32dcce-8f25-4d7d-bd86-fa16c63737f1
  spec:
    image: ""
    logEnabled: true
    version: v1.2.2
  status:
    conditions:
    - status: Unknown
      type: Completed
    - lastUpdateTime: "2024-05-17T06:40:00Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2024-05-17T06:41:04Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2024-05-17T06:43:48Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2024-05-17T06:59:11Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2024-05-17T07:13:55Z"
      status: "True"
      type: SystemServicesUpgraded
    - status: Unknown
      type: NodesUpgraded
    imageID: harvester-system/hvst-upgrade-hjw2p
    nodeStatuses:
      harvester-dc-0:
        state: Pre-drained
      harvester-dc-1:
        state: Images preloaded
      harvester-dc-2:
        state: Images preloaded
      harvester-dc-3:
        state: Images preloaded
    previousVersion: v1.2.1
    repoInfo: |
      release:
          harvester: v1.2.2
          harvesterChart: 1.2.2
          os: Harvester v1.2.2
          kubernetes: v1.26.15+rke2r1
          rancher: v2.8.2
          monitoringChart: 103.0.3+up45.31.1
          minUpgradableVersion: v1.2.1
    upgradeLog: hvst-upgrade-hjw2p-upgradelog
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-04-25T13:56:45Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 31
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-nszm9
    namespace: harvester-system
    resourceVersion: "217748009"
    uid: 23f16027-63e5-464d-8332-22b187f9ee14
  spec:
    image: ""
    logEnabled: true
    version: v1.1.2
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T15:35:16Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-04-25T14:00:58Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-04-25T14:04:05Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-04-25T14:36:40Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-04-25T14:47:01Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-04-25T15:35:16Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/harvester-iso-wjfvr
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
    previousVersion: v1.1.1
    repoInfo: |
      release:
          harvester: v1.1.2
          harvesterChart: 1.1.2
          os: Harvester v1.1.2
          kubernetes: v1.24.11+rke2r1
          rancher: v2.6.11
          monitoringChart: 100.1.0+up19.0.3
          minUpgradableVersion: v1.1.0
- apiVersion: harvesterhci.io/v1beta1
  kind: Upgrade
  metadata:
    annotations:
      harvesterhci.io/replica-replenishment-wait-interval: "600"
    creationTimestamp: "2023-10-27T06:56:30Z"
    finalizers:
    - wrangler.cattle.io/harvester-upgrade-controller
    generateName: hvst-upgrade-
    generation: 41
    labels:
      harvesterhci.io/read-message: "true"
      harvesterhci.io/upgradeState: Succeeded
    name: hvst-upgrade-wmxxs
    namespace: harvester-system
    resourceVersion: "599919539"
    uid: 4534471a-5492-45bc-9d30-4147ce7c1cb4
  spec:
    image: ""
    logEnabled: true
    version: v1.2.1
  status:
    conditions:
    - lastUpdateTime: "2023-10-27T09:47:15Z"
      status: "True"
      type: Completed
    - lastUpdateTime: "2023-10-27T06:56:57Z"
      status: "True"
      type: LogReady
    - lastUpdateTime: "2023-10-27T06:58:48Z"
      status: "True"
      type: ImageReady
    - lastUpdateTime: "2023-10-27T07:02:28Z"
      status: "True"
      type: RepoReady
    - lastUpdateTime: "2023-10-27T07:10:38Z"
      status: "True"
      type: NodesPrepared
    - lastUpdateTime: "2023-10-27T07:11:49Z"
      status: "True"
      type: SystemServicesUpgraded
    - lastUpdateTime: "2023-10-27T09:47:15Z"
      status: "True"
      type: NodesUpgraded
    imageID: harvester-system/hvst-upgrade-wmxxs
    nodeStatuses:
      harvester-dc-0:
        state: Succeeded
      harvester-dc-1:
        state: Succeeded
      harvester-dc-2:
        state: Succeeded
      harvester-dc-3:
        state: Succeeded
    previousVersion: v1.2.0
    repoInfo: |
      release:
          harvester: v1.2.1
          harvesterChart: 1.2.1
          os: Harvester v1.2.1
          kubernetes: v1.25.9+rke2r1
          rancher: v2.7.5
          monitoringChart: 102.0.0+up40.1.2
          minUpgradableVersion: v1.1.2
kind: List
metadata:
  resourceVersion: ""

apiVersion: v1
items:
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: additional-ca
    resourceVersion: "8451"
    uid: 8b9493e6-d17c-4e12-a0c4-e52bf1bea2f2
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: 1.1.9
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: api-ui-version
    resourceVersion: "8448"
    uid: 969dc386-9401-403d-9373-10efa8fa73b7
  status: {}
  value: 1.1.9
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: auto-disk-provision-paths
    resourceVersion: "8469"
    uid: e7556dd1-af28-4b8e-a3a9-b9d289d9e7d0
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 29
    name: backup-target
    resourceVersion: "339358074"
    uid: 79121a2e-fef6-422f-8510-866eaf575847
  status:
    conditions:
    - lastUpdateTime: "2023-11-27T09:58:03Z"
      status: "True"
      type: configured
  value: '{"type":"s3","endpoint":"https://s3-east-internal.buct.edu.cn:9000","accessKeyId":"","secretAccessKey":"","bucketName":"vm-backups","bucketRegion":"bj","cert":"","virtualHostedStyle":false}'
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/hash: 30cff57968b3b923f62143d6987eb1eb077ca0d8061b21f0db658123
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 3
    name: cluster-registration-url
    resourceVersion: "14406"
    uid: 5b7684de-3054-4cec-beec-6d6265d89bd4
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T07:23:29Z"
      status: "True"
      type: configured
  value: https://rancher.buct.edu.cn/v3/import/td7vhx6j6sg6qxp88g5flhzzsdndz96xjrl7zvd2nzq2hwkp5b788t_c-m-76mpd8w7.yaml
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/upgrade-patched: v1.2.2
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: containerd-registry
    resourceVersion: "599960369"
    uid: de373ce8-27d0-4004-8947-609e0f1f0315
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"driver.longhorn.io":{"volumeSnapshotClassName":"longhorn-snapshot","backupVolumeSnapshotClassName":"longhorn"}}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: csi-driver-config
    resourceVersion: "8454"
    uid: a7fd421c-5e71-42c0-ad13-2e268f6b1a16
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: longhorn
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: default-storage-class
    resourceVersion: "8460"
    uid: a6f605fb-8789-4570-aa68-ee8870da980c
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "120"
  kind: Setting
  metadata:
    creationTimestamp: "2023-09-09T07:25:51Z"
    generation: 1
    name: default-vm-termination-grace-period-seconds
    resourceVersion: "217801286"
    uid: e488e5d9-a0f7-41ff-922e-2b7c1dfb593f
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"harvester-cloud-provider":">=0.0.1 <0.3.0","harvester-csi-provider":">=0.0.1
    <0.3.0"}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 3
    name: harvester-csi-ccm-versions
    resourceVersion: "217801288"
    uid: f9926d02-cf54-4b64-8662-dc13109361eb
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: http-proxy
    resourceVersion: "8425"
    uid: 322b0282-a4b9-4262-8095-e3e9185bc4a1
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: info
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: log-level
    resourceVersion: "8457"
    uid: 9c4e74df-9f16-4249-9fd0-da88f8932763
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-09-09T07:25:51Z"
    generation: 1
    name: ntp-servers
    resourceVersion: "217801281"
    uid: 3cf5d828-79b5-484e-abdb-4482a1d961c9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"cpu":1600,"memory":150,"storage":200}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: overcommit-config
    resourceVersion: "8423"
    uid: 1f0fdcf1-1e50-4fc0-abd3-646f3fe0d018
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: release-download-url
    resourceVersion: "8459"
    uid: 4ba73310-bcd8-4c2c-84c6-af147d4058fb
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: dev
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 5
    name: server-version
    resourceVersion: "599949818"
    uid: d1cbce86-a2d6-4bb4-b73a-caf15630c82b
  status: {}
  value: v1.2.2
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    annotations:
      harvesterhci.io/hash: d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac5b3e42f
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ssl-certificates
    resourceVersion: "8498"
    uid: 514e3aeb-5e62-4b0c-ac33-cb8a2a399182
  status:
    conditions:
    - lastUpdateTime: "2023-04-25T07:17:36Z"
      status: "False"
      type: configured
- apiVersion: harvesterhci.io/v1beta1
  default: '{}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ssl-parameters
    resourceVersion: "8463"
    uid: da2eeb6e-0340-4aaf-85a1-e1079b2e9d51
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: storage-network
    resourceVersion: "8455"
    uid: 9ee25fee-5bbb-4a31-9663-204d04c090db
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"repository":"rancher/support-bundle-kit","tag":"v0.0.37","imagePullPolicy":"IfNotPresent"}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 66
    name: support-bundle-image
    resourceVersion: "599953683"
    uid: 9554026e-23ce-45b8-935a-e4146a7278a9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: support-bundle-namespaces
    resourceVersion: "8461"
    uid: 886c947d-dd8e-46a9-a934-ecd0282900c9
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "10"
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: support-bundle-timeout
    resourceVersion: "8442"
    uid: 357d73d0-7e6d-4b60-a3bf-962a7d483b49
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester-ui/dashboard/release-harvester-v1.2/index.html
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ui-index
    resourceVersion: "217801285"
    uid: 237c20d9-09c8-4d43-a38b-12f5860cd02d
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: /usr/share/harvester/harvester
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ui-path
    resourceVersion: "8429"
    uid: 4e838361-c370-48fd-b86d-f3b3bb799c43
  status: {}
  value: /usr/share/harvester/harvester
- apiVersion: harvesterhci.io/v1beta1
  default: https://releases.rancher.com/harvester-ui/plugin/harvester-release-harvester-v1.2/harvester-release-harvester-v1.2.umd.min.js
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 2
    name: ui-plugin-index
    resourceVersion: "217801290"
    uid: 8270d656-502f-4f73-b778-5b7ae82687ca
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: auto
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: ui-source
    resourceVersion: "8470"
    uid: 839901a8-0944-474c-ac21-7469438719ef
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgradable-versions
    resourceVersion: "8462"
    uid: 84d85c47-5057-415b-8ba1-7040a49ae09e
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: "true"
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgrade-checker-enabled
    resourceVersion: "8432"
    uid: 2d2b4c3d-8a23-40df-bab6-edc60a62368c
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: https://harvester-upgrade-responder.rancher.io/v1/checkupgrade
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: upgrade-checker-url
    resourceVersion: "8437"
    uid: fa4e35fe-d384-48df-ba77-8cb12c175b9e
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: vip-pools
    resourceVersion: "8453"
    uid: 4b1dedb7-9e36-4a07-a37f-606db62aac86
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: '{"enable":true,"period":300}'
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: vm-force-reset-policy
    resourceVersion: "8452"
    uid: 5fae2e04-cd67-4214-848d-9cd21b1b1fd4
  status: {}
- apiVersion: harvesterhci.io/v1beta1
  default: longhorn
  kind: Setting
  metadata:
    creationTimestamp: "2023-04-25T07:17:35Z"
    generation: 1
    name: volume-snapshot-class
    resourceVersion: "8450"
    uid: 9616bfa4-5c17-42eb-84ce-32d84238401c
  status: {}
kind: List
metadata:
  resourceVersion: ""

[Vicente-Cheng]

Hi @W0n9, Could you also check the suggestion from @starbops

It might be related to certificates expiration. Perhaps you could check if https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311 works for you. Thanks.

[W0n9]

Hi @Vicente-Cheng , I have tried https://github.com/harvester/harvester/issues/3863#issuecomment-1539681311 in my 2 long-running harvester cluster, and it works. Thanks

[bk201]

cert rotate is implemented in v1.3.0, @starbops can you help add this (https://docs.harvesterhci.io/v1.3/upgrade/v1-1-2-to-v1-2-0#7-upgrade-stuck-in-the-upgrading-system-service-state) and seeder one to the known issue sections of 1.2.2 upgrade page? Thanks.