search

metal-stack / gardener-extension-provider-metal

Implementation of the gardener-extension-controller for metal-stack
MIT License
24 stars 11 forks source link

health-container on metallb-controller fails #240

Closed mwennrich closed 2 years ago

mwennrich commented 2 years ago 
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Normal   Pulled   46m                   kubelet  Successfully pulled image "bitnami/kubectl:latest" in 350.534804ms
  Normal   Pulled   37m                   kubelet  Successfully pulled image "bitnami/kubectl:latest" in 358.420789ms
  Warning  BackOff  35m (x24 over 31h)    kubelet  Back-off restarting failed container
  Normal   Pulled   35m                   kubelet  Successfully pulled image "bitnami/kubectl:latest" in 285.201474ms
  Normal   Pulling  8m18s (x57 over 31h)  kubelet  Pulling image "bitnami/kubectl:latest"
  Normal   Created  8m17s (x57 over 31h)  kubelet  Created container health
  Normal   Started  8m17s (x57 over 31h)  kubelet  Started container health
  Normal   Pulled   8m17s                 kubelet  Successfully pulled image "bitnami/kubectl:latest" in 340.201031m

kubectl client is pretty slow:

nobody@controller-57dc9bdd7c-bnns8:/$ time kubectl get cm health     
I0223 15:40:51.538053    3933 request.go:665] Waited for 1.141841233s due to client-side throttling, not priority and fairness, request: GET:https://api.example.com:443/apis/extensions/v1beta1?timeout=32s
NAME     DATA   AGE
health   2      29d

real    0m7.676s

Caching fails:

nobody@controller-57dc9bdd7c-bnns8:/$ kubectl get cm health -v=4
I0223 15:40:08.579675    3768 merged_client_builder.go:163] Using in-cluster namespace
I0223 15:40:08.579944    3768 merged_client_builder.go:121] Using in-cluster configuration
I0223 15:40:08.613402    3768 cached_discovery.go:130] failed to write cache to /.kube/cache/discovery/api.example.com_443/servergroups.json due to mkdir /.kube/cache: read-only file system
I0223 15:40:08.617519    3768 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/api.example.com_443/coordination.k8s.io/v1beta1/serverresources.json due to mkdir /.kube/cache: read-only file system
I0223 15:40:08.617543    3768 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/api.example.com_443/scheduling.k8s.io/v1beta1/serverresources.json due to mkdir /.kube/cache: read-only file system
I0223 15:40:08.617587    3768 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/api.example.com_443/batch/v1/serverresources.json due to mkdir /.kube/cache: read-only file system
I0223 15:40:08.617631    3768 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/api.example.com_443/scheduling.k8s.io/v1/serverresources.json due to mkdir /.kube/cache: read-only file system
I0223 15:40:08.617744    3768 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/api.example.com_443/acme.cert-manager.io/v1/serverresources.json due to mkdir /.kube/cache: read-only file system
(...)

Shoud probably set --cache-dir "/tmp/.kube/cache" or something