pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.24k stars 499 forks source link

1.5.1 can not upgrade tidb 6.5 to 7.5 due to not support alibaba oss volume #5453

Open imhun opened 11 months ago

imhun commented 11 months ago

Bug Report

What version of Kubernetes are you using?

1.24.6 What version of TiDB Operator are you using?

1.5.1 What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

image

What did you do?

helm upgrade install tidb-operator 1.5.1 edit tidbcluster.yaml to 7.5

pd and tiflash was updated from 6.5 to 7.,5 but tikv was not updated,still 6.5

and tidb-controller-manager error log: 2023-12-12T09:35:29.254709747+08:00 W1212 01:35:29.254643 1 sync_volume_status.go:80] skip to observe volume status for pod test-tidb/test-tikv-1, because failed to get actual volumes: storageclass.storage.k8s.io "oss" not found

2023-12-12T09:35:29.254726266+08:00 W1212 01:35:29.254665 1 sync_volume_status.go:80] skip to observe volume status for pod test-tidb/test-tikv-0, because failed to get actual volumes: storageclass.storage.k8s.io "oss" not found

2023-12-12T09:35:29.254728926+08:00 W1212 01:35:29.254672 1 sync_volume_status.go:80] skip to observe volume status for pod test-tidb/test-tikv-2, because failed to get actual volumes: storageclass.storage.k8s.io "oss" not found What did you expect to see? tikv upgrade to 7.5,or all tidb rollback to 6.5 What did you see instead? pd and tiflash is 7.5,but tikv and tidb is still 6.5

csuzhangxc commented 11 months ago

The above is just some warn logs, are there more logs?

imhun commented 11 months ago

no more logs,upgrade is stuck because the pvc check failed it should ignore this error,because the old version tikv pod was already running it should check it when create new pod instead of checking what already exists

csuzhangxc commented 11 months ago

but the continue wouldn't block reconciliation,

https://github.com/pingcap/tidb-operator/blob/v1.5.1/pkg/manager/volumes/sync_volume_status.go#L79-L82