search

banzaicloud / banzai-charts

Curated list of Banzai Cloud Helm charts used by the Pipeline Platform
Apache License 2.0
368 stars 283 forks source link

kube-metrics-adapter FailedDiscoveryCheck #1004

Open Arnold1 opened 4 years ago

Arnold1 commented 4 years ago

Describe the bug

hi, I have some issue with kube-metrics-adapter and FailedDiscoveryCheck. how to fix it?

$ kubectl get apiservice
NAME                                   SERVICE                            AVAILABLE                      AGE
v1.                                    Local                              True                           27d
v1.apps                                Local                              True                           27d
v1.authentication.k8s.io               Local                              True                           27d
v1.authorization.k8s.io                Local                              True                           27d
v1.autoscaling                         Local                              True                           27d
v1.batch                               Local                              True                           27d
v1.coordination.k8s.io                 Local                              True                           27d
v1.networking.k8s.io                   Local                              True                           27d
v1.rbac.authorization.k8s.io           Local                              True                           27d
v1.scheduling.k8s.io                   Local                              True                           27d
v1.storage.k8s.io                      Local                              True                           27d
v1alpha1.authentication.istio.io       Local                              True                           26d
v1alpha1.certmanager.k8s.io            Local                              True                           27d
v1alpha1.rbac.istio.io                 Local                              True                           27d
v1alpha2.config.istio.io               Local                              True                           27d
v1alpha3.networking.istio.io           Local                              True                           27d
v1beta1.admissionregistration.k8s.io   Local                              True                           27d
v1beta1.apiextensions.k8s.io           Local                              True                           27d
v1beta1.apps                           Local                              True                           27d
v1beta1.authentication.k8s.io          Local                              True                           27d
v1beta1.authorization.k8s.io           Local                              True                           27d
v1beta1.batch                          Local                              True                           27d
v1beta1.certificates.k8s.io            Local                              True                           27d
v1beta1.coordination.k8s.io            Local                              True                           27d
v1beta1.custom.metrics.k8s.io          kube-system/kube-metrics-adapter   False (FailedDiscoveryCheck)   23d
v1beta1.events.k8s.io                  Local                              True                           27d
v1beta1.extensions                     Local                              True                           27d
v1beta1.external.metrics.k8s.io        kube-system/kube-metrics-adapter   False (FailedDiscoveryCheck)   20h
v1beta1.metrics.k8s.io                 kube-system/metrics-server         True                           27d
v1beta1.networking.k8s.io              Local                              True                           27d
v1beta1.node.k8s.io                    Local                              True                           27d
v1beta1.policy                         Local                              True                           27d
v1beta1.rbac.authorization.k8s.io      Local                              True                           27d
v1beta1.scheduling.k8s.io              Local                              True                           27d
v1beta1.security.istio.io              Local                              True                           27d
v1beta1.storage.k8s.io                 Local                              True                           27d
v1beta2.apps                           Local                              True                           27d
v2beta1.autoscaling                    Local                              True                           27d
v2beta2.autoscaling                    Local                              True                           27d

$ kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.custom.metrics.k8s.io"},"spec":{"group":"custom.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"kube-metrics-adapter","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: "2019-12-21T12:49:33Z"
  labels:
    app: prometheus-adapter
    chart: prometheus-adapter-1.4.0
    heritage: Tiller
    release: custom-metrics
  name: v1beta1.custom.metrics.k8s.io
  resourceVersion: "28643935"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
  uid: cf90678e-b876-42c0-bd7b-3bcdba2b49e1
spec:
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: kube-metrics-adapter
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-01-13T15:05:14Z"
    message: 'failing or missing response from https://100.64.254.89:443/apis/custom.metrics.k8s.io/v1beta1:
      Get https://100.64.254.89:443/apis/custom.metrics.k8s.io/v1beta1: net/http:
      request canceled while waiting for connection (Client.Timeout exceeded while
      awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

$ kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"name":"v1beta1.external.metrics.k8s.io"},"spec":{"group":"external.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"kube-metrics-adapter","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: "2020-01-12T22:13:26Z"
  name: v1beta1.external.metrics.k8s.io
  resourceVersion: "28643936"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.external.metrics.k8s.io
  uid: 27480d61-e080-411e-bcfe-56c26f6f4a8a
spec:
  group: external.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: kube-metrics-adapter
    namespace: kube-system
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2020-01-13T15:05:14Z"
    message: 'failing or missing response from https://100.64.254.89:443/apis/external.metrics.k8s.io/v1beta1:
      Get https://100.64.254.89:443/apis/external.metrics.k8s.io/v1beta1: net/http:
      request canceled while waiting for connection (Client.Timeout exceeded while
      awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

$ helm ls
NAME            REVISION    UPDATED                     STATUS      CHART                       APP VERSION NAMESPACE   
custom-metrics  1           Sat Dec 21 07:49:33 2019    DEPLOYED    prometheus-adapter-1.4.0    v0.5.0      istio-system
istio           14          Thu Jan  2 11:18:35 2020    DEPLOYED    istio-1.4.0                 1.4.0       istio-system
istio-init      1           Tue Dec 17 10:22:09 2019    DEPLOYED    istio-init-1.4.0            1.4.0       istio-system
kube2iam        1           Sat Jan  4 14:47:21 2020    DEPLOYED    kube2iam-2.1.0              0.10.7      kube-system 
metrics-server  1           Tue Dec 17 13:28:16 2019    DEPLOYED    metrics-server-2.8.8        0.3.5       kube-system

Steps to reproduce the issue:

$ helm install --name kube-metrics-adapter stable/kube-metrics-adapter

Expected behavior

no error

Screenshots

no

Additional context

tuananhnguyen-ct commented 4 years ago

I had the same problem, but with apiservice connecting to port 8443 (default service.internalPort) at pod, fixed with creating a firewall rule to allow master to connect via port 8443 (on private cluster, only traffic via 443/10250 is allowed by default, ref https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules).

sancyx commented 4 years ago

There was a similar issue https://github.com/banzaicloud/banzai-charts/issues/1060, where enabling host network solved the problem, which is available in version 0.1.3. @Arnold1 is this problem still occurring? In case it still occurs, could you pease describe k8s version , cloud provider, network plugin?