search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.24k stars 499 forks source link

PD pod unschedulable with scheduler extender #180

Closed tennix closed 5 years ago

tennix commented 6 years ago

From #170, when deploying a TiDB cluster in DinD with Local PV, even we specify WaitForFirstConsumer binding mode for local volume. The PV and PVC still may get bound before the pod is scheduled.

Previously we don't have scheduler extender, and all pods can be scheduled. But now when adding scheduler extender, PD pods are not allowed to schedule on the same node. So when PV and PVC are bound before pod scheduling, the pod can actually only be scheduled on the node that PV belongs to. But the HA predicate would refuse to schedule on that node, causing pod unschedulable.

This may be related to local volume scheduling bug which uses affinity and anti-affinity rules. It is fixed in Kubernetes v1.12.

tennix commented 5 years ago

This seems affect all location based persistent volumes scheduling. For example GKE regional persistent disk as in #241

tlightsky commented 5 years ago

i meet this issue in OSX DinD env too

weekface commented 5 years ago

@tlightsky Are you using manifests/local-dind/dind-cluster-v1.10.sh to build your DinD env?

gregwebs commented 5 years ago

@tlightsky what is the K8s version?

tlightsky commented 5 years ago

@weekface yes, i used version v1.0.0-beta.1, then i switched to master, then clean/up again, this problem disappear

weekface commented 5 years ago

this should be fixed in the newer version.