search

kelseyhightower / kubernetes-the-hard-way

Bootstrap Kubernetes the hard way. No scripts.
Apache License 2.0
41.3k stars 14.13k forks source link

Kube-scheduler restarting #642

Open aswinkm-tc opened 3 years ago

aswinkm-tc commented 3 years ago

After following the docs, I'm stuck at doc 08. The kube-scheduler keeps restarting throwing the following error

Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.236966    4642 registry.go:150] Registering EvenPodsSpread predicate and priority function
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.237743    4642 registry.go:150] Registering EvenPodsSpread predicate and priority function
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238217    4642 flags.go:33] FLAG: --add-dir-header="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238401    4642 flags.go:33] FLAG: --address="0.0.0.0"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238535    4642 flags.go:33] FLAG: --algorithm-provider=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238555    4642 flags.go:33] FLAG: --alsologtostderr="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238566    4642 flags.go:33] FLAG: --authentication-kubeconfig=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238572    4642 flags.go:33] FLAG: --authentication-skip-lookup="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238582    4642 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="10s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238590    4642 flags.go:33] FLAG: --authentication-tolerate-lookup-failure="true"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238596    4642 flags.go:33] FLAG: --authorization-always-allow-paths="[/healthz]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238613    4642 flags.go:33] FLAG: --authorization-kubeconfig=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238619    4642 flags.go:33] FLAG: --authorization-webhook-cache-authorized-ttl="10s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238626    4642 flags.go:33] FLAG: --authorization-webhook-cache-unauthorized-ttl="10s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238632    4642 flags.go:33] FLAG: --bind-address="0.0.0.0"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238641    4642 flags.go:33] FLAG: --cert-dir=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238647    4642 flags.go:33] FLAG: --client-ca-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238652    4642 flags.go:33] FLAG: --config="/etc/kubernetes/config/kube-scheduler.yaml"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238659    4642 flags.go:33] FLAG: --contention-profiling="true"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238665    4642 flags.go:33] FLAG: --feature-gates=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238674    4642 flags.go:33] FLAG: --hard-pod-affinity-symmetric-weight="1"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238682    4642 flags.go:33] FLAG: --help="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238689    4642 flags.go:33] FLAG: --http2-max-streams-per-connection="0"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238702    4642 flags.go:33] FLAG: --kube-api-burst="100"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238708    4642 flags.go:33] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238715    4642 flags.go:33] FLAG: --kube-api-qps="50"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238724    4642 flags.go:33] FLAG: --kubeconfig=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238729    4642 flags.go:33] FLAG: --leader-elect="true"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238735    4642 flags.go:33] FLAG: --leader-elect-lease-duration="15s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238741    4642 flags.go:33] FLAG: --leader-elect-renew-deadline="10s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238747    4642 flags.go:33] FLAG: --leader-elect-resource-lock="endpointsleases"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238753    4642 flags.go:33] FLAG: --leader-elect-resource-name="kube-scheduler"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238761    4642 flags.go:33] FLAG: --leader-elect-resource-namespace="kube-system"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238767    4642 flags.go:33] FLAG: --leader-elect-retry-period="2s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238773    4642 flags.go:33] FLAG: --lock-object-name="kube-scheduler"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238779    4642 flags.go:33] FLAG: --lock-object-namespace="kube-system"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238785    4642 flags.go:33] FLAG: --log-backtrace-at=":0"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238795    4642 flags.go:33] FLAG: --log-dir=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238802    4642 flags.go:33] FLAG: --log-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238808    4642 flags.go:33] FLAG: --log-file-max-size="1800"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238813    4642 flags.go:33] FLAG: --log-flush-frequency="5s"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238819    4642 flags.go:33] FLAG: --logtostderr="true"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238825    4642 flags.go:33] FLAG: --master=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238836    4642 flags.go:33] FLAG: --policy-config-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238841    4642 flags.go:33] FLAG: --policy-configmap=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238847    4642 flags.go:33] FLAG: --policy-configmap-namespace="kube-system"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238853    4642 flags.go:33] FLAG: --port="10251"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238859    4642 flags.go:33] FLAG: --profiling="true"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238865    4642 flags.go:33] FLAG: --requestheader-allowed-names="[]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238875    4642 flags.go:33] FLAG: --requestheader-client-ca-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238882    4642 flags.go:33] FLAG: --requestheader-extra-headers-prefix="[x-remote-extra-]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238892    4642 flags.go:33] FLAG: --requestheader-group-headers="[x-remote-group]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238899    4642 flags.go:33] FLAG: --requestheader-username-headers="[x-remote-user]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238908    4642 flags.go:33] FLAG: --scheduler-name="default-scheduler"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238915    4642 flags.go:33] FLAG: --secure-port="10259"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238921    4642 flags.go:33] FLAG: --show-hidden-metrics-for-version=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238927    4642 flags.go:33] FLAG: --skip-headers="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238932    4642 flags.go:33] FLAG: --skip-log-headers="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238938    4642 flags.go:33] FLAG: --stderrthreshold="2"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238948    4642 flags.go:33] FLAG: --tls-cert-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238954    4642 flags.go:33] FLAG: --tls-cipher-suites="[]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238960    4642 flags.go:33] FLAG: --tls-min-version=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238966    4642 flags.go:33] FLAG: --tls-private-key-file=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238972    4642 flags.go:33] FLAG: --tls-sni-cert-key="[]"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238979    4642 flags.go:33] FLAG: --use-legacy-policy-config="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238985    4642 flags.go:33] FLAG: --v="2"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.238991    4642 flags.go:33] FLAG: --version="false"
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.239000    4642 flags.go:33] FLAG: --vmodule=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.239006    4642 flags.go:33] FLAG: --write-config-to=""
Mar 16 15:09:47 controller-0 kube-scheduler[4642]: I0316 15:09:47.840126    4642 serving.go:313] Generated self-signed cert in-memory
Mar 16 15:09:48 controller-0 kube-scheduler[4642]: W0316 15:09:48.321201    4642 authentication.go:268] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
Mar 16 15:09:48 controller-0 kube-scheduler[4642]: W0316 15:09:48.321953    4642 authentication.go:292] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
Mar 16 15:09:48 controller-0 kube-scheduler[4642]: W0316 15:09:48.322078    4642 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
Mar 16 15:09:48 controller-0 kube-scheduler[4642]: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Mar 16 15:09:48 controller-0 systemd[1]: kube-scheduler.service: Main process exited, code=exited, status=1/FAILURE
Mar 16 15:09:48 controller-0 systemd[1]: kube-scheduler.service: Failed with result 'exit-code'.

Version:

kube-scheduler --version
I0316 15:10:12.493318    4751 registry.go:150] Registering EvenPodsSpread predicate and priority function
I0316 15:10:12.493386    4751 registry.go:150] Registering EvenPodsSpread predicate and priority function
Kubernetes v1.18.6

Components Status

kubectl get componentstatuses --kubeconfig admin.kubeconfig
NAME                 STATUS      MESSAGE                                                                                     ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused   
controller-manager   Healthy     ok                                                                                          
etcd-0               Healthy     {"health":"true"}                                                                           
etcd-1               Healthy     {"health":"true"}                                                                           
etcd-2               Healthy     {"health":"true"}

Systemd unit file

cat /etc/systemd/system/kube-scheduler.service 
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
  --config=/etc/kubernetes/config/kube-scheduler.yaml \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Kube schduler Config file

cat /etc/kubernetes/config/kube-scheduler.yaml 
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true

I tried adding the authentication-kubeconfig and authorization-kubeconfig flags by pointing to the /var/lib/kubernetes/kube-scheduler.kubeconfig file but it was throwing another error

failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable