scheduler: MountVolume.NodeAffinity check failed for volume "" : No matching NodeSelectorTerms

[xiaojingchen]

k8s version:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• I found a tikv pod pending in ContainerCreating when deploy tidb cluster

  stability-cluster1-tikv-2                       0/2     ContainerCreating   0          14h   <none>          172.16.4.155   <none>

• exec kubectl describe po stability-cluster1-tikv-2 -n stability-cluster1

  Events:
  Type     Reason       Age                       From                   Message
  ----     ------       ----                      ----                   -------
  Warning  FailedMount  4m51s (x525488 over 14h)  kubelet, 172.16.4.155  MountVolume.NodeAffinity check failed for volume "local-pv-284c9de3" : No matching NodeSelectorTerms

• found the pod's pvc and pv

  tikv-stability-cluster1-tikv-2   Bound    local-pv-284c9de3   1476Gi     RWO            local-storage   14h

• get pv info

  [tidb@172.16.4.155 ~]$ kubectl get pv  local-pv-284c9de3 -o yaml
  apiVersion: v1
  kind: PersistentVolume
  metadata:
  annotations:
  pv.kubernetes.io/bound-by-controller: "yes"
  pv.kubernetes.io/provisioned-by: local-volume-provisioner-172.16.4.156-0ea7c9c2-5a1b-11e9-b9e5-525400cb79ec
  tidb.pingcap.com/pod-name: stability-cluster1-tikv-2
  creationTimestamp: 2019-04-09T05:39:34Z
  finalizers:
  - kubernetes.io/pv-protection
  labels:
  app.kubernetes.io/component: tikv
  app.kubernetes.io/instance: stability-cluster1
  app.kubernetes.io/managed-by: tidb-operator
  app.kubernetes.io/name: tidb-cluster
  app.kubernetes.io/namespace: stability-cluster1
  tidb.pingcap.com/cluster-id: "6677872351424076801"
  name: local-pv-284c9de3
  resourceVersion: "205144"
  selfLink: /api/v1/persistentvolumes/local-pv-284c9de3
  uid: dad8fb51-5a89-11e9-b9e5-525400cb79ec
  spec:
  accessModes:
  - ReadWriteOnce
  capacity:
  storage: 1476Gi
  claimRef:
  apiVersion: v1
  kind: PersistentVolumeClaim
  name: tikv-stability-cluster1-tikv-2
  namespace: stability-cluster1
  resourceVersion: "204509"
  uid: f3331988-5ac3-11e9-9bf5-525400cb79ec
  local:
  path: /mnt/disks/local-pv02
  nodeAffinity:
  required:
    nodeSelectorTerms:
    - matchExpressions:
      - key: kubernetes.io/hostname
        operator: In
        values:
        - 172.16.4.156
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  status:
  phase: Bound

  then i found the pod was scheduled on 172.16.4.155, but the pv is on 172.16.4.156. I think the the pod scheduled in node without considering PV requirements.

[xiaojingchen]

found the issue which explains the bug: https://github.com/kubernetes/kubernetes/issues/69928 PTAL

[weekface]

This is fixed in v1.14: https://github.com/kubernetes/kubernetes/pull/70026

@onlymellb @tennix We may need to upgrade to v1.14.

[aylei]

This is fixed in v1.14: kubernetes/kubernetes#70026

@onlymellb @tennix We may need to upgrade to v1.14.

Can we port this to 1.12? Most of hosted-kubernetes vendors using 1.12 now.

[xiaojingchen]

I wonder why this bug didn't appear before, but it recurs frequently now. We've been using v1.12.2.

[xiaojingchen]

the reason is that the kubernetes cluster have two tidb-scheduler. we should provide a mechanism to ensure that there is only one tidb-operator in k8s cluster.

[weekface]

308