TiDB operator cannot update configuration for TiKV

Bug Report

What version of Kubernetes are you using?

1.28.1

What version of TiDB Operator are you using?

1.6.0

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

standard

What's the status of the TiDB cluster pods?

What did you do?

We tried to change the storage.engine option of TiKV, via the spec.tikv.config property. But we found that after changing storage.engine from raft-kv to partitioned-raft-kv, the TiKV keeps using raft-kv as the storage engine.

Install TiDB cluster by applying the CR.

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
name: basic
spec:
version: v8.1.0
timezone: UTC
pvReclaimPolicy: Retain
enableDynamicConfiguration: true
configUpdateStrategy: RollingUpdate
discovery: {}
helper:
image: alpine:3.16.0
pd:
baseImage: pingcap/pd
maxFailoverCount: 0
replicas: 1
requests:
  storage: "1Gi"
config: {}
tikv:
baseImage: pingcap/tikv
maxFailoverCount: 0
evictLeaderTimeout: 1m
replicas: 1
requests:
  storage: "1Gi"
config:
  storage:
    # In basic examples, we set this to avoid using too much storage.
    reserve-space: "0MB"
    engine: "raft-kv"
  rocksdb:
    # In basic examples, we set this to avoid the following error in some Kubernetes clusters:
    # "the maximum number of open file descriptors is too small, got 1024, expect greater or equal to 82920"
    max-open-files: 256
  raftdb:
    max-open-files: 256
tidb:
baseImage: pingcap/tidb
maxFailoverCount: 0
replicas: 1
service:
  type: ClusterIP
config: {}

Change the spec.tikv.config.storage.engine to partitioned-raft-kv

Use mysqlsh to query the current configuration, and still see the engine is raft-kv

+------+----------------------------------------------------------------+----------------+---------+
| Type | Instance                                                       | Name           | Value    |
+------+----------------------------------------------------------------+----------------+---------+
| tikv | advanced-tidb-tikv-2.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
| tikv | advanced-tidb-tikv-1.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
| tikv | advanced-tidb-tikv-0.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
+------+----------------------------------------------------------------+----------------+---------+

What did you expect to see? Storage engine should be updated, or the config change should be rejected explicitly.

What did you see instead? Although the Pods are rolling restarted with the updated ConfigMap, the actual configuration being used by the TiKV is not updated. We suspect that the updated configuration gets lost when the TiDB merges the configuration.

pingcap / tidb-operator

TiDB operator cannot update configuration for TiKV #5729

Bug Report