search

Huang-Wei / sample-scheduler-extender

a sample to showcase how to create a k8s scheduler extender
Apache License 2.0
56 stars 48 forks source link

Error: failed to acquire lease kube-system/kube-scheduler #2

Closed kbudakovskiy closed 5 years ago

kbudakovskiy commented 5 years ago

Hi, thanks for your work! I try to create a custom schedule by your example: https://developer.ibm.com/articles/creating-a-custom-kube-scheduler/ I've got a main(default) scheduler and I want to have a extra custom scheduler for particular pods. I use prebuild scheduler container with kubeconfig for autorization:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: asa-i-scheduler-type1
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: asa-i-scheduler-type1-as-kube-scheduler
subjects:
- kind: ServiceAccount
  name: asa-i-scheduler-type1
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: asa-i-scheduler-type1-volume-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:volume-scheduler
subjects:
- kind: ServiceAccount
  name: asa-i-scheduler-type1
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: random-scheduler-config-role-binding
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: asa-i-scheduler-type1
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    component: scheduler
    tier: control-plane
  name: asa-i-scheduler-type1
  namespace: kube-system
spec:
  selector:
    matchLabels:
      component: scheduler
      tier: control-plane
  replicas: 1
  template: 
    metadata:
      labels:
        component: scheduler
        tier: control-plane
        version: second
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - stage-k8s-gate-test-n1.k8s
                - stage-k8s-gate-test-n2.k8s
      serviceAccountName: asa-i-scheduler-type1
      imagePullSecrets:
      - name: regcred
      containers:
      - name: kube-custom-scheduler
        image: registry.office/kubernetes/custom-scheduler:empty
        command:
        - /usr/local/bin/kube-scheduler
        - --address=0.0.0.0
        - --leader-elect=false
        - --scheduler-name=asa-i-scheduler-type1
        - --config=/root/scheduler-extender-config.yaml
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10251
          initialDelaySeconds: 15
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10251
        resources:
          requests:
            cpu: '0.1'
        securityContext:
          privileged: false
        volumeMounts: []
      - name:  kube-scheduler-plugin
        env:
        - name: DEBUG
          value: "true"
        image: registry.office/kubernetes/custom-plugin:empty
        ports:
        - containerPort: 8888
      hostNetwork: false
      hostPID: false
      volumes: []

This policy:

{
    "kind" : "Policy",
    "apiVersion" : "v1",
    "extenders" : [{
        "urlPrefix": "http://127.0.0.1:8888/",
        "filterVerb": "filter",
        "prioritizeVerb": "prioritize",
        "weight": 1,
        "enableHttps": false
    }]
}

And:

apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
leaderelection:  "false"
clientConnection:
  kubeconfig: "/var/run/kubernetes/scheduler.kubeconfig"
algorithmSource:
  policy:
    file:
      path: "/root/scheduler-extender-policy.json"
      # YAML not supported yet
      # https://github.com/kubernetes/kubernetes/issues/75852

Also I add modify REST API. added some markers. for ex:

func Filter(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
    fmt.Println("Filter has been queried")

This is my deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pause
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pause
  template:
    metadata:
      labels:
        app: pause
    spec:
      schedulerName: asa-i-scheduler-type1
      containers:
      - name: pause
        image: k8s.gcr.io/pause:3.1

But when deployed it pods fall in Pending:

pause-7b79d4869b-9slzc                  0/1     Pending   0          10s
pause-7b79d4869b-9xvp9                  0/1     Pending   0          10s

And REST log is empty.

I1009 08:58:09.568650       1 flags.go:33] FLAG: --address="0.0.0.0"
I1009 08:58:09.568688       1 flags.go:33] FLAG: --algorithm-provider=""
I1009 08:58:09.568692       1 flags.go:33] FLAG: --alsologtostderr="false"
I1009 08:58:09.568696       1 flags.go:33] FLAG: --authentication-kubeconfig=""
I1009 08:58:09.568701       1 flags.go:33] FLAG: --authentication-skip-lookup="false"
I1009 08:58:09.568706       1 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
I1009 08:58:09.568709       1 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="true"
I1009 08:58:09.568712       1 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
I1009 08:58:09.568719       1 flags.go:33] FLAG: --authorization-kubeconfig=""
I1009 08:58:09.568722       1 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
I1009 08:58:09.568725       1 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
I1009 08:58:09.568728       1 flags.go:33] FLAG: --bind-address="0.0.0.0"
I1009 08:58:09.568732       1 flags.go:33] FLAG: --cert-dir=""
I1009 08:58:09.568734       1 flags.go:33] FLAG: --client-ca-file=""
I1009 08:58:09.568737       1 flags.go:33] FLAG: --config="/root/scheduler-extender-config.yaml"
I1009 08:58:09.568740       1 flags.go:33] FLAG: --contention-profiling="false"
I1009 08:58:09.568743       1 flags.go:33] FLAG: --feature-gates=""
I1009 08:58:09.568747       1 flags.go:33] FLAG: --hard-pod-affinity-symmetric-weight="1"
I1009 08:58:09.568751       1 flags.go:33] FLAG: --help="false"
I1009 08:58:09.568754       1 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
I1009 08:58:09.568758       1 flags.go:33] FLAG: --kube-api-burst="100"
I1009 08:58:09.568761       1 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I1009 08:58:09.568764       1 flags.go:33] FLAG: --kube-api-qps="50"
I1009 08:58:09.568773       1 flags.go:33] FLAG: --kubeconfig=""
I1009 08:58:09.568776       1 flags.go:33] FLAG: --leader-elect="false"
I1009 08:58:09.568779       1 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
I1009 08:58:09.568781       1 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
I1009 08:58:09.568784       1 flags.go:33] FLAG: --leader-elect-resource-lock="endpoints"
I1009 08:58:09.568787       1 flags.go:33] FLAG: --leader-elect-retry-period="2s"
I1009 08:58:09.568790       1 flags.go:33] FLAG: --lock-object-name="kube-scheduler"
I1009 08:58:09.568793       1 flags.go:33] FLAG: --lock-object-namespace="kube-system"
I1009 08:58:09.568796       1 flags.go:33] FLAG: --log-backtrace-at=":0"
I1009 08:58:09.568800       1 flags.go:33] FLAG: --log-dir=""
I1009 08:58:09.568803       1 flags.go:33] FLAG: --log-file=""
I1009 08:58:09.568806       1 flags.go:33] FLAG: --log-file-max-size="1800"
I1009 08:58:09.568809       1 flags.go:33] FLAG: --log-flush-frequency="5s"
I1009 08:58:09.568812       1 flags.go:33] FLAG: --logtostderr="true"
I1009 08:58:09.568814       1 flags.go:33] FLAG: --master=""
I1009 08:58:09.568817       1 flags.go:33] FLAG: --policy-config-file=""
I1009 08:58:09.568820       1 flags.go:33] FLAG: --policy-configmap=""
I1009 08:58:09.568822       1 flags.go:33] FLAG: --policy-configmap-namespace="kube-system"
I1009 08:58:09.568825       1 flags.go:33] FLAG: --port="10251"
I1009 08:58:09.568828       1 flags.go:33] FLAG: --profiling="false"
I1009 08:58:09.568831       1 flags.go:33] FLAG: --requestheader-allowed-names="[]"
I1009 08:58:09.568834       1 flags.go:33] FLAG: --requestheader-client-ca-file=""
I1009 08:58:09.568837       1 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
I1009 08:58:09.568842       1 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
I1009 08:58:09.568845       1 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
I1009 08:58:09.568854       1 flags.go:33] FLAG: --scheduler-name="asa-i-scheduler-type1"
I1009 08:58:09.568857       1 flags.go:33] FLAG: --secure-port="10259"
I1009 08:58:09.568874       1 flags.go:33] FLAG: --skip-headers="false"
I1009 08:58:09.568876       1 flags.go:33] FLAG: --skip-log-headers="false"
I1009 08:58:09.568879       1 flags.go:33] FLAG: --stderrthreshold="2"
I1009 08:58:09.568883       1 flags.go:33] FLAG: --tls-cert-file=""
I1009 08:58:09.568886       1 flags.go:33] FLAG: --tls-cipher-suites="[]"
I1009 08:58:09.568889       1 flags.go:33] FLAG: --tls-min-version=""
I1009 08:58:09.568891       1 flags.go:33] FLAG: --tls-private-key-file=""
I1009 08:58:09.568894       1 flags.go:33] FLAG: --tls-sni-cert-key="[]"
I1009 08:58:09.568898       1 flags.go:33] FLAG: --use-legacy-policy-config="false"
I1009 08:58:09.568901       1 flags.go:33] FLAG: --v="5"
I1009 08:58:09.568904       1 flags.go:33] FLAG: --version="false"
I1009 08:58:09.568908       1 flags.go:33] FLAG: --vmodule=""
I1009 08:58:09.568911       1 flags.go:33] FLAG: --write-config-to=""
I1009 08:58:09.837184       1 serving.go:319] Generated self-signed cert in-memory
I1009 08:58:10.055305       1 server.go:142] Version: v1.15.1
I1009 08:58:10.055356       1 defaults.go:87] TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory
I1009 08:58:10.055371       1 server.go:161] Starting Kubernetes Scheduler version v1.15.1
I1009 08:58:10.061839       1 factory.go:355] Creating scheduler from configuration: {{ } [] [] [{http://127.0.0.1:8888/ filter  prioritize 1  false <nil> 0s false [] false}] 0 false}
I1009 08:58:10.061889       1 factory.go:364] Using predicates from algorithm provider 'DefaultProvider'
I1009 08:58:10.061896       1 factory.go:379] Using priorities from algorithm provider 'DefaultProvider'
I1009 08:58:10.061900       1 factory.go:397] Creating extender with config {URLPrefix:http://127.0.0.1:8888/ FilterVerb:filter PreemptVerb: PrioritizeVerb:prioritize Weight:1 BindVerb: EnableHTTPS:false TLSConfig:<nil> HTTPTimeout:0s NodeCacheCapable:false ManagedResources:[] Ignorable:false}
I1009 08:58:10.061920       1 factory.go:433] Creating scheduler with fit predicates 'map[CheckNodeUnschedulable:{} CheckVolumeBinding:{} GeneralPredicates:{} MatchInterPodAffinity:{} MaxAzureDiskVolumeCount:{} MaxCSIVolumeCountPred:{} MaxEBSVolumeCount:{} MaxGCEPDVolumeCount:{} NoDiskConflict:{} NoVolumeZoneConflict:{} PodToleratesNodeTaints:{}]' and priority functions 'map[BalancedResourceAllocation:{} ImageLocalityPriority:{} InterPodAffinityPriority:{} LeastRequestedPriority:{} NodeAffinityPriority:{} NodePreferAvoidPodsPriority:{} SelectorSpreadPriority:{} TaintTolerationPriority:{}]'
I1009 08:58:10.062183       1 healthz.go:107] Installing healthz checkers:"leaderElection"
W1009 08:58:10.062566       1 authorization.go:47] Authorization is disabled
W1009 08:58:10.062575       1 authentication.go:55] Authentication is disabled
I1009 08:58:10.062588       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I1009 08:58:10.062600       1 healthz.go:107] Installing healthz checkers:"leaderElection"
I1009 08:58:10.063045       1 secure_serving.go:116] Serving securely on [::]:10259
I1009 08:58:10.063319       1 reflector.go:122] Starting reflector *v1.StorageClass (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063337       1 reflector.go:160] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063449       1 reflector.go:122] Starting reflector *v1.StatefulSet (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063459       1 reflector.go:160] Listing and watching *v1.StatefulSet from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063527       1 reflector.go:122] Starting reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063550       1 reflector.go:160] Listing and watching *v1.Service from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063690       1 reflector.go:122] Starting reflector *v1.ReplicaSet (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063700       1 reflector.go:160] Listing and watching *v1.ReplicaSet from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063927       1 reflector.go:122] Starting reflector *v1.PersistentVolumeClaim (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063938       1 reflector.go:160] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063967       1 reflector.go:122] Starting reflector *v1.ReplicationController (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.063975       1 reflector.go:160] Listing and watching *v1.ReplicationController from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064153       1 reflector.go:122] Starting reflector *v1.Pod (0s) from k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:226
I1009 08:58:10.064160       1 reflector.go:160] Listing and watching *v1.Pod from k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:226
I1009 08:58:10.064165       1 reflector.go:122] Starting reflector *v1.PersistentVolume (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064172       1 reflector.go:160] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064394       1 reflector.go:122] Starting reflector *v1beta1.PodDisruptionBudget (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064404       1 reflector.go:160] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064457       1 reflector.go:122] Starting reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.064477       1 reflector.go:160] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:133
I1009 08:58:10.104270       1 node_tree.go:93] Added node stage-k8s-board-n3.k8s.office in group  to NodeTree
I1009 08:58:10.104353       1 node_tree.go:93] Added node stage-k8s-gate-test-n1.k8s.office in group  to NodeTree
I1009 08:58:10.104414       1 node_tree.go:93] Added node stage-k8s-gate-test-n2.k8s.office in group  to NodeTree
I1009 08:58:10.104468       1 node_tree.go:93] Added node stage-k8s-board-n4.k8s.office in group  to NodeTree
I1009 08:58:10.104553       1 node_tree.go:93] Added node stage-k8s-board-n5.k8s.office in group  to NodeTree
I1009 08:58:10.104616       1 node_tree.go:93] Added node stage-k8s-gate-test-ilb1.k8s.office in group  to NodeTree
I1009 08:58:10.104634       1 node_tree.go:93] Added node dev-gate-log1.dev.gate.office in group  to NodeTree
I1009 08:58:10.104663       1 node_tree.go:93] Added node stage-k8s-board-crr2.k8s.office in group  to NodeTree
I1009 08:58:10.104686       1 node_tree.go:93] Added node stage-k8s-board-crr3.k8s.office in group  to NodeTree
I1009 08:58:10.104704       1 node_tree.go:93] Added node stage-k8s-board-ilb2.k8s.office in group  to NodeTree
I1009 08:58:10.104722       1 node_tree.go:93] Added node stage-k8s-board-n2.k8s.office in group  to NodeTree
I1009 08:58:10.104771       1 node_tree.go:93] Added node stage-k8s-gate-test-ilb2.k8s.office in group  to NodeTree
I1009 08:58:10.104786       1 node_tree.go:93] Added node dev-gate-log2.dev.gate.office in group  to NodeTree
I1009 08:58:10.104802       1 node_tree.go:93] Added node stage-k8s-board-crr1.k8s.office in group  to NodeTree
I1009 08:58:10.104827       1 node_tree.go:93] Added node stage-k8s-board-log1.stage.k8s.office in group  to NodeTree
I1009 08:58:10.104847       1 node_tree.go:93] Added node stage-k8s-board-log2.stage.k8s.office in group  to NodeTree
I1009 08:58:10.104873       1 node_tree.go:93] Added node stage-k8s-board-m1.k8s.office in group  to NodeTree
I1009 08:58:10.104896       1 node_tree.go:93] Added node dev-gate-log3.dev.gate.office in group  to NodeTree
I1009 08:58:10.104937       1 node_tree.go:93] Added node stage-k8s-board-crr4.k8s.office in group  to NodeTree
I1009 08:58:10.104961       1 node_tree.go:93] Added node stage-k8s-board-ilb1.k8s.office in group  to NodeTree
I1009 08:58:10.104978       1 node_tree.go:93] Added node stage-k8s-board-m2.k8s.office in group  to NodeTree
I1009 08:58:10.104994       1 node_tree.go:93] Added node stage-k8s-board-n1.k8s.office in group  to NodeTree
I1009 08:58:10.163229       1 shared_informer.go:176] caches populated
I1009 08:58:10.263345       1 shared_informer.go:176] caches populated
I1009 08:58:10.363499       1 shared_informer.go:176] caches populated
I1009 08:58:10.463675       1 shared_informer.go:176] caches populated
I1009 08:58:10.563837       1 shared_informer.go:176] caches populated
I1009 08:58:10.664032       1 shared_informer.go:176] caches populated
I1009 08:58:10.764278       1 shared_informer.go:176] caches populated
I1009 08:58:10.864518       1 shared_informer.go:176] caches populated
I1009 08:58:10.964741       1 shared_informer.go:176] caches populated
I1009 08:58:10.964924       1 leaderelection.go:235] attempting to acquire leader lease  kube-system/kube-scheduler...
I1009 08:58:10.969915       1 leaderelection.go:344] lock is held by stage-k8s-board-m2.k8s.office_9e668b49-72c8-4e88-837d-ea4ef50f20c7 and has not yet expired
I1009 08:58:10.969939       1 leaderelection.go:240] failed to acquire lease kube-system/kube-scheduler
I1009 08:58:12.721224       1 pathrecorder.go:240] kube-scheduler: "/healthz" satisfied by exact match
I1009 08:58:12.721376       1 wrap.go:47] GET /healthz: (156.39µs) 200 [kube-probe/1.15 10.210.81.246:11494]
I1009 08:58:14.415976       1 leaderelection.go:344] lock is held by stage-k8s-board-m2.k8s.office_9e668b49-72c8-4e88-837d-ea4ef50f20c7 and has not yet expired
I1009 08:58:14.416003       1 leaderelection.go:240] failed to acquire lease kube-system/kube-scheduler
I1009 08:58:16.972017       1 leaderelection.go:344] lock is held by stage-k8s-board-m2.k8s.office_9e668b49-72c8-4e88-837d-ea4ef50f20c7 and has not yet expired
I

If I remove
- --config=/root/scheduler-extender-config.yaml Than everithing works well. but with default shaduling policy.

Why when I start use policy with scheduler-extender-config.yaml custom scheduler wants to become a leader and doesn't work?

Huang-Wei commented 5 years ago

It's because the flag of --leader-elect has been deprecated, it will be honored without the presence of --config. In other words, if you use --config, the scheduler will only respect the "leader-elect" value from the Config object.

And I checked the code, if no leaderElection fields are specified, it will default to true:

https://github.com/kubernetes/kubernetes/blob/4e7ef9ad3650310e5ea3fce3dbe65779437e3716/pkg/scheduler/apis/config/v1alpha1/defaults.go#L95

https://github.com/kubernetes/kubernetes/blob/2e52d5c3311d3dcfd042e81570ef138645de529a/staging/src/k8s.io/component-base/config/v1alpha1/defaults.go#L49-L51

This explains why you hit this issue.

So for now, you can work it around by updating scheduler-extender-config.yaml to:

apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/run/kubernetes/scheduler.kubeconfig"
leaderElection:
  leaderElect: false
algorithmSource:
  policy:
    file:
      path: "/root/config/scheduler-extender-policy.json"
      # YAML not supported yet
      # https://github.com/kubernetes/kubernetes/issues/75852

Give a try and let me know if it works.

kbudakovskiy commented 5 years ago

Many thanks, now the problem is gone, but pods still Pending. It looks like a custom scheduler doesn't see pods in schedule queue.

kbudakovskiy commented 5 years ago

I added SchedulerName string `json:"schedulerName" And everything works well. Thank you