kubernetes / ingress-nginx

Ingress NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.43k stars 8.24k forks source link

my ingress controller metrics api gives few part of metrics #7025

Closed nwaistd closed 2 years ago

nwaistd commented 3 years ago

I deployed my ingress controller like this on AWS EKS

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx

---
# Source: ingress-nginx/templates/controller-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
---
# Source: ingress-nginx/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
---
# Source: ingress-nginx/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
  name: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ''
    resources:
      - configmaps
      - pods
      - secrets
      - endpoints
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - services
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - networking.k8s.io   # k8s 1.14+
    resources:
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ''
    resources:
      - configmaps
    resourceNames:
      - ingress-controller-leader-nginx
    verbs:
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ''
    resources:
      - endpoints
    verbs:
      - create
      - get
      - update
  - apiGroups:
      - ''
    resources:
      - events
    verbs:
      - create
      - patch
---
# Source: ingress-nginx/templates/controller-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx
subjects:
  - kind: ServiceAccount
    name: ingress-nginx
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/controller-service-webhook.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission
  namespace: ingress-nginx
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60'
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ...
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
#    - name: http
    #      port: 80
    #      protocol: TCP
    #      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:
      priorityClassName: system
      nodeSelector:
        kubernetes.io/os: linux
        env: kube-system
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: k8s.gcr.io/ingress-nginx/controller:v0.40.0@sha256:b954d8ff1466eb236162c644bd64e9027a212c82b484cbe47cc21da45fe8bc59
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --publish-service=ingress-nginx/ingress-nginx-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=ingress-nginx/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: metrics
              containerPort: 10254
              protocol: TCP
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
            - name: webhook
              containerPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
# Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
# before changing this value, check the required kubernetes version
# https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission
  namespace: ingress-nginx
webhooks:
  - name: validate.nginx.ingress.kubernetes.io
    rules:
      - apiGroups:
          - extensions
          - networking.k8s.io
        apiVersions:
          - v1beta1
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission
        path: /networking/v1beta1/ingresses
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
rules:
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
    verbs:
      - get
      - update
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-create
  annotations:
    helm.sh/hook: pre-install,pre-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
spec:
  template:
    metadata:
      name: ingress-nginx-admission-create
      labels:
        helm.sh/chart: ingress-nginx-3.4.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.40.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: create
          image: docker.io/jettech/kube-webhook-certgen:v1.3.0
          imagePullPolicy: IfNotPresent
          args:
            - create
            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.ingress-nginx.svc
            - --namespace=ingress-nginx
            - --secret-name=ingress-nginx-admission
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: ingress-nginx-admission-patch
  annotations:
    helm.sh/hook: post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-3.4.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.40.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
spec:
  template:
    metadata:
      name: ingress-nginx-admission-patch
      labels:
        helm.sh/chart: ingress-nginx-3.4.0
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/version: 0.40.0
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: admission-webhook
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: patch
          image: docker.io/jettech/kube-webhook-certgen:v1.3.0
          imagePullPolicy: IfNotPresent
          args:
            - patch
            - --webhook-name=ingress-nginx-admission
            - --namespace=ingress-nginx
            - --patch-mutating=false
            - --secret-name=ingress-nginx-admission
            - --patch-failure-policy=Fail
          env:
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      restartPolicy: OnFailure
      serviceAccountName: ingress-nginx-admission
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ''
    resources:
      - secrets
    verbs:
      - get
      - create
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ingress-nginx-admission
subjects:
  - kind: ServiceAccount
    name: ingress-nginx-admission
    namespace: ingress-nginx
---
# Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ingress-nginx-admission
  annotations:
    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
  labels:
    helm.sh/chart: ingress-nginx-2.0.3
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: admission-webhook
  namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller-metrics
  namespace: ingress-nginx
  labels:
    metrics: ingress-nginx
spec:
  ports:
    - name: metrics
      port: 10254
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/component: controller

When I run curl localhost:10254/metrics in the ingress-controller-pod, it seems to show some part of all metrics. for example,

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.7971e-05
go_gc_duration_seconds{quantile="0.25"} 4.687e-05
go_gc_duration_seconds{quantile="0.5"} 6.1087e-05
go_gc_duration_seconds{quantile="0.75"} 7.0657e-05
go_gc_duration_seconds{quantile="1"} 0.004151713
go_gc_duration_seconds_sum 0.008315222
go_gc_duration_seconds_count 62
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 90
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.2731432e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 5.57795824e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.532268e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.703594e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 2.5696427128104918e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.9352e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.2731432e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.8816128e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.6326656e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 60433
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.1730048e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5142784e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6177296561107635e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.764027e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 176528
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 327680
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 2.257432e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 397876
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.96608e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.96608e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.5318272e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP nginx_ingress_controller_config_hash Running configuration hash actually running
# TYPE nginx_ingress_controller_config_hash gauge
nginx_ingress_controller_config_hash{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.7535943280805745e+18
# HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful
# TYPE nginx_ingress_controller_config_last_reload_successful gauge
nginx_ingress_controller_config_last_reload_successful{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1
# HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge
nginx_ingress_controller_config_last_reload_successful_timestamp_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.617726872e+09
# HELP nginx_ingress_controller_leader_election_status Gauge reporting status of the leader election, 0 indicates follower, 1 indicates leader. 'name' is the string used to identify the lease
# TYPE nginx_ingress_controller_leader_election_status gauge
nginx_ingress_controller_leader_election_status{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",name="ingress-controller-leader-nginx"} 1
# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting}
# TYPE nginx_ingress_controller_nginx_process_connections gauge
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="active"} 1
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="reading"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="waiting"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="writing"} 1
# HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled}
# TYPE nginx_ingress_controller_nginx_process_connections_total counter
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="accepted"} 713
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="handled"} 713
# HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds
# TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter
nginx_ingress_controller_nginx_process_cpu_seconds_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.7900000000000005
# HELP nginx_ingress_controller_nginx_process_num_procs number of processes
# TYPE nginx_ingress_controller_nginx_process_num_procs gauge
nginx_ingress_controller_nginx_process_num_procs{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 9
# HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge
nginx_ingress_controller_nginx_process_oldest_start_time_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.61772687e+09
# HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read
# TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter
nginx_ingress_controller_nginx_process_read_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 0
# HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests
# TYPE nginx_ingress_controller_nginx_process_requests_total counter
nginx_ingress_controller_nginx_process_requests_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 739
# HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge
nginx_ingress_controller_nginx_process_resident_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.25009664e+08
# HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge
nginx_ingress_controller_nginx_process_virtual_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.155986944e+09
# HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written
# TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter
nginx_ingress_controller_nginx_process_write_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 4096
# HELP nginx_ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire.\n           An example to check if this certificate will expire in 10 days is: "nginx_ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))"
# TYPE nginx_ingress_controller_ssl_expire_time_seconds gauge
nginx_ingress_controller_ssl_expire_time_seconds{class="nginx",host="_",namespace="ingress-nginx"} 1.649262871e+09
# HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations
# TYPE nginx_ingress_controller_success counter
nginx_ingress_controller_success{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 6.51
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 39
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 6.0248064e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61772687031e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.61356288e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 110
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

So, most of dashboard are empty.

Screen Shot 2021-04-07 at 2 34 44 AM

What am I missing?

please helpme

/triage support

k8s-ci-robot commented 3 years ago

@nwaistd: The label(s) triage/support cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7025): > > > > >when i deployed my ingress controller like this on AWS EKS >``` >apiVersion: v1 >kind: Namespace >metadata: > name: ingress-nginx > labels: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > >--- ># Source: ingress-nginx/templates/controller-serviceaccount.yaml >apiVersion: v1 >kind: ServiceAccount >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/controller-configmap.yaml >apiVersion: v1 >kind: ConfigMap >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx-controller > namespace: ingress-nginx >data: >--- ># Source: ingress-nginx/templates/clusterrole.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: ClusterRole >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > name: ingress-nginx >rules: > - apiGroups: > - '' > resources: > - configmaps > - endpoints > - nodes > - pods > - secrets > verbs: > - list > - watch > - apiGroups: > - '' > resources: > - nodes > verbs: > - get > - apiGroups: > - '' > resources: > - services > verbs: > - get > - list > - update > - watch > - apiGroups: > - extensions > - networking.k8s.io # k8s 1.14+ > resources: > - ingresses > verbs: > - get > - list > - watch > - apiGroups: > - '' > resources: > - events > verbs: > - create > - patch > - apiGroups: > - extensions > - networking.k8s.io # k8s 1.14+ > resources: > - ingresses/status > verbs: > - update > - apiGroups: > - networking.k8s.io # k8s 1.14+ > resources: > - ingressclasses > verbs: > - get > - list > - watch >--- ># Source: ingress-nginx/templates/clusterrolebinding.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: ClusterRoleBinding >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > name: ingress-nginx >roleRef: > apiGroup: rbac.authorization.k8s.io > kind: ClusterRole > name: ingress-nginx >subjects: > - kind: ServiceAccount > name: ingress-nginx > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/controller-role.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: Role >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx > namespace: ingress-nginx >rules: > - apiGroups: > - '' > resources: > - namespaces > verbs: > - get > - apiGroups: > - '' > resources: > - configmaps > - pods > - secrets > - endpoints > verbs: > - get > - list > - watch > - apiGroups: > - '' > resources: > - services > verbs: > - get > - list > - update > - watch > - apiGroups: > - extensions > - networking.k8s.io # k8s 1.14+ > resources: > - ingresses > verbs: > - get > - list > - watch > - apiGroups: > - extensions > - networking.k8s.io # k8s 1.14+ > resources: > - ingresses/status > verbs: > - update > - apiGroups: > - networking.k8s.io # k8s 1.14+ > resources: > - ingressclasses > verbs: > - get > - list > - watch > - apiGroups: > - '' > resources: > - configmaps > resourceNames: > - ingress-controller-leader-nginx > verbs: > - get > - update > - apiGroups: > - '' > resources: > - configmaps > verbs: > - create > - apiGroups: > - '' > resources: > - endpoints > verbs: > - create > - get > - update > - apiGroups: > - '' > resources: > - events > verbs: > - create > - patch >--- ># Source: ingress-nginx/templates/controller-rolebinding.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: RoleBinding >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx > namespace: ingress-nginx >roleRef: > apiGroup: rbac.authorization.k8s.io > kind: Role > name: ingress-nginx >subjects: > - kind: ServiceAccount > name: ingress-nginx > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/controller-service-webhook.yaml >apiVersion: v1 >kind: Service >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx-controller-admission > namespace: ingress-nginx >spec: > type: ClusterIP > ports: > - name: https-webhook > port: 443 > targetPort: webhook > selector: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/component: controller >--- ># Source: ingress-nginx/templates/controller-service.yaml >apiVersion: v1 >kind: Service >metadata: > annotations: > service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http > service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '60' > service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' > service.beta.kubernetes.io/aws-load-balancer-type: nlb > service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ... > service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx-controller > namespace: ingress-nginx >spec: > type: LoadBalancer > externalTrafficPolicy: Local > ports: ># - name: http > # port: 80 > # protocol: TCP > # targetPort: http > - name: https > port: 443 > protocol: TCP > targetPort: http > selector: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/component: controller >--- ># Source: ingress-nginx/templates/controller-deployment.yaml >apiVersion: apps/v1 >kind: Deployment >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: controller > name: ingress-nginx-controller > namespace: ingress-nginx >spec: > selector: > matchLabels: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/component: controller > revisionHistoryLimit: 10 > minReadySeconds: 0 > template: > metadata: > labels: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/component: controller > spec: > priorityClassName: system > nodeSelector: > kubernetes.io/os: linux > env: kube-system > dnsPolicy: ClusterFirst > containers: > - name: controller > image: k8s.gcr.io/ingress-nginx/controller:v0.40.0@sha256:b954d8ff1466eb236162c644bd64e9027a212c82b484cbe47cc21da45fe8bc59 > imagePullPolicy: IfNotPresent > lifecycle: > preStop: > exec: > command: > - /wait-shutdown > args: > - /nginx-ingress-controller > - --publish-service=ingress-nginx/ingress-nginx-controller > - --election-id=ingress-controller-leader > - --ingress-class=nginx > - --configmap=ingress-nginx/ingress-nginx-controller > - --validating-webhook=:8443 > - --validating-webhook-certificate=/usr/local/certificates/cert > - --validating-webhook-key=/usr/local/certificates/key > securityContext: > capabilities: > drop: > - ALL > add: > - NET_BIND_SERVICE > runAsUser: 101 > allowPrivilegeEscalation: true > env: > - name: POD_NAME > valueFrom: > fieldRef: > fieldPath: metadata.name > - name: POD_NAMESPACE > valueFrom: > fieldRef: > fieldPath: metadata.namespace > - name: LD_PRELOAD > value: /usr/local/lib/libmimalloc.so > livenessProbe: > httpGet: > path: /healthz > port: 10254 > scheme: HTTP > initialDelaySeconds: 10 > periodSeconds: 10 > timeoutSeconds: 1 > successThreshold: 1 > failureThreshold: 3 > readinessProbe: > httpGet: > path: /healthz > port: 10254 > scheme: HTTP > initialDelaySeconds: 10 > periodSeconds: 10 > timeoutSeconds: 1 > successThreshold: 1 > failureThreshold: 3 > ports: > - name: metrics > containerPort: 10254 > protocol: TCP > - name: http > containerPort: 80 > protocol: TCP > - name: https > containerPort: 443 > protocol: TCP > - name: webhook > containerPort: 8443 > protocol: TCP > volumeMounts: > - name: webhook-cert > mountPath: /usr/local/certificates/ > readOnly: true > resources: > requests: > cpu: 100m > memory: 90Mi > serviceAccountName: ingress-nginx > terminationGracePeriodSeconds: 300 > volumes: > - name: webhook-cert > secret: > secretName: ingress-nginx-admission >--- ># Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml ># before changing this value, check the required kubernetes version ># https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites >apiVersion: admissionregistration.k8s.io/v1 >kind: ValidatingWebhookConfiguration >metadata: > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > name: ingress-nginx-admission > namespace: ingress-nginx >webhooks: > - name: validate.nginx.ingress.kubernetes.io > rules: > - apiGroups: > - extensions > - networking.k8s.io > apiVersions: > - v1beta1 > - v1 > operations: > - CREATE > - UPDATE > resources: > - ingresses > failurePolicy: Fail > sideEffects: None > admissionReviewVersions: > - v1 > - v1beta1 > clientConfig: > service: > namespace: ingress-nginx > name: ingress-nginx-controller-admission > path: /networking/v1beta1/ingresses >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml >apiVersion: v1 >kind: ServiceAccount >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: ClusterRole >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook >rules: > - apiGroups: > - admissionregistration.k8s.io > resources: > - validatingwebhookconfigurations > verbs: > - get > - update >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: ClusterRoleBinding >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook >roleRef: > apiGroup: rbac.authorization.k8s.io > kind: ClusterRole > name: ingress-nginx-admission >subjects: > - kind: ServiceAccount > name: ingress-nginx-admission > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: Role >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >rules: > - apiGroups: > - '' > resources: > - secrets > verbs: > - get > - create >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: RoleBinding >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >roleRef: > apiGroup: rbac.authorization.k8s.io > kind: Role > name: ingress-nginx-admission >subjects: > - kind: ServiceAccount > name: ingress-nginx-admission > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml >apiVersion: batch/v1 >kind: Job >metadata: > name: ingress-nginx-admission-create > annotations: > helm.sh/hook: pre-install,pre-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >spec: > template: > metadata: > name: ingress-nginx-admission-create > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > spec: > nodeSelector: > kubernetes.io/os: linux > containers: > - name: create > image: docker.io/jettech/kube-webhook-certgen:v1.3.0 > imagePullPolicy: IfNotPresent > args: > - create > - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.ingress-nginx.svc > - --namespace=ingress-nginx > - --secret-name=ingress-nginx-admission > env: > - name: POD_NAMESPACE > valueFrom: > fieldRef: > fieldPath: metadata.namespace > restartPolicy: OnFailure > serviceAccountName: ingress-nginx-admission > securityContext: > runAsNonRoot: true > runAsUser: 2000 >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml >apiVersion: batch/v1 >kind: Job >metadata: > name: ingress-nginx-admission-patch > annotations: > helm.sh/hook: post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >spec: > template: > metadata: > name: ingress-nginx-admission-patch > labels: > helm.sh/chart: ingress-nginx-3.4.0 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.40.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > spec: > nodeSelector: > kubernetes.io/os: linux > containers: > - name: patch > image: docker.io/jettech/kube-webhook-certgen:v1.3.0 > imagePullPolicy: IfNotPresent > args: > - patch > - --webhook-name=ingress-nginx-admission > - --namespace=ingress-nginx > - --patch-mutating=false > - --secret-name=ingress-nginx-admission > - --patch-failure-policy=Fail > env: > - name: POD_NAMESPACE > valueFrom: > fieldRef: > fieldPath: metadata.namespace > restartPolicy: OnFailure > serviceAccountName: ingress-nginx-admission > securityContext: > runAsNonRoot: true > runAsUser: 2000 >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: Role >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-2.0.3 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.32.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >rules: > - apiGroups: > - '' > resources: > - secrets > verbs: > - get > - create >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml >apiVersion: rbac.authorization.k8s.io/v1 >kind: RoleBinding >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-2.0.3 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.32.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >roleRef: > apiGroup: rbac.authorization.k8s.io > kind: Role > name: ingress-nginx-admission >subjects: > - kind: ServiceAccount > name: ingress-nginx-admission > namespace: ingress-nginx >--- ># Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml >apiVersion: v1 >kind: ServiceAccount >metadata: > name: ingress-nginx-admission > annotations: > helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade > helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded > labels: > helm.sh/chart: ingress-nginx-2.0.3 > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/version: 0.32.0 > app.kubernetes.io/managed-by: Helm > app.kubernetes.io/component: admission-webhook > namespace: ingress-nginx >--- >apiVersion: v1 >kind: Service >metadata: > name: ingress-nginx-controller-metrics > namespace: ingress-nginx > labels: > metrics: ingress-nginx >spec: > ports: > - name: metrics > port: 10254 > selector: > app.kubernetes.io/name: ingress-nginx > app.kubernetes.io/instance: ingress-nginx > app.kubernetes.io/component: controller >``` > > >When I run `bash-5.0$ curl localhost:10254/metrics` in the ingress-controller-pod, it seems -to shoe some part of all metrics. >for example, >``` ># HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. ># TYPE go_gc_duration_seconds summary >go_gc_duration_seconds{quantile="0"} 1.7971e-05 >go_gc_duration_seconds{quantile="0.25"} 4.687e-05 >go_gc_duration_seconds{quantile="0.5"} 6.1087e-05 >go_gc_duration_seconds{quantile="0.75"} 7.0657e-05 >go_gc_duration_seconds{quantile="1"} 0.004151713 >go_gc_duration_seconds_sum 0.008315222 >go_gc_duration_seconds_count 62 ># HELP go_goroutines Number of goroutines that currently exist. ># TYPE go_goroutines gauge >go_goroutines 90 ># HELP go_info Information about the Go environment. ># TYPE go_info gauge >go_info{version="go1.15.2"} 1 ># HELP go_memstats_alloc_bytes Number of bytes allocated and still in use. ># TYPE go_memstats_alloc_bytes gauge >go_memstats_alloc_bytes 1.2731432e+07 ># HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed. ># TYPE go_memstats_alloc_bytes_total counter >go_memstats_alloc_bytes_total 5.57795824e+08 ># HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table. ># TYPE go_memstats_buck_hash_sys_bytes gauge >go_memstats_buck_hash_sys_bytes 1.532268e+06 ># HELP go_memstats_frees_total Total number of frees. ># TYPE go_memstats_frees_total counter >go_memstats_frees_total 4.703594e+06 ># HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started. ># TYPE go_memstats_gc_cpu_fraction gauge >go_memstats_gc_cpu_fraction 2.5696427128104918e-05 ># HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata. ># TYPE go_memstats_gc_sys_bytes gauge >go_memstats_gc_sys_bytes 5.9352e+06 ># HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use. ># TYPE go_memstats_heap_alloc_bytes gauge >go_memstats_heap_alloc_bytes 1.2731432e+07 ># HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used. ># TYPE go_memstats_heap_idle_bytes gauge >go_memstats_heap_idle_bytes 4.8816128e+07 ># HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use. ># TYPE go_memstats_heap_inuse_bytes gauge >go_memstats_heap_inuse_bytes 1.6326656e+07 ># HELP go_memstats_heap_objects Number of allocated objects. ># TYPE go_memstats_heap_objects gauge >go_memstats_heap_objects 60433 ># HELP go_memstats_heap_released_bytes Number of heap bytes released to OS. ># TYPE go_memstats_heap_released_bytes gauge >go_memstats_heap_released_bytes 4.1730048e+07 ># HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system. ># TYPE go_memstats_heap_sys_bytes gauge >go_memstats_heap_sys_bytes 6.5142784e+07 ># HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection. ># TYPE go_memstats_last_gc_time_seconds gauge >go_memstats_last_gc_time_seconds 1.6177296561107635e+09 ># HELP go_memstats_lookups_total Total number of pointer lookups. ># TYPE go_memstats_lookups_total counter >go_memstats_lookups_total 0 ># HELP go_memstats_mallocs_total Total number of mallocs. ># TYPE go_memstats_mallocs_total counter >go_memstats_mallocs_total 4.764027e+06 ># HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures. ># TYPE go_memstats_mcache_inuse_bytes gauge >go_memstats_mcache_inuse_bytes 3472 ># HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system. ># TYPE go_memstats_mcache_sys_bytes gauge >go_memstats_mcache_sys_bytes 16384 ># HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures. ># TYPE go_memstats_mspan_inuse_bytes gauge >go_memstats_mspan_inuse_bytes 176528 ># HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system. ># TYPE go_memstats_mspan_sys_bytes gauge >go_memstats_mspan_sys_bytes 327680 ># HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place. ># TYPE go_memstats_next_gc_bytes gauge >go_memstats_next_gc_bytes 2.257432e+07 ># HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations. ># TYPE go_memstats_other_sys_bytes gauge >go_memstats_other_sys_bytes 397876 ># HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator. ># TYPE go_memstats_stack_inuse_bytes gauge >go_memstats_stack_inuse_bytes 1.96608e+06 ># HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator. ># TYPE go_memstats_stack_sys_bytes gauge >go_memstats_stack_sys_bytes 1.96608e+06 ># HELP go_memstats_sys_bytes Number of bytes obtained from system. ># TYPE go_memstats_sys_bytes gauge >go_memstats_sys_bytes 7.5318272e+07 ># HELP go_threads Number of OS threads created. ># TYPE go_threads gauge >go_threads 15 ># HELP nginx_ingress_controller_config_hash Running configuration hash actually running ># TYPE nginx_ingress_controller_config_hash gauge >nginx_ingress_controller_config_hash{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.7535943280805745e+18 ># HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful ># TYPE nginx_ingress_controller_config_last_reload_successful gauge >nginx_ingress_controller_config_last_reload_successful{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1 ># HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload. ># TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge >nginx_ingress_controller_config_last_reload_successful_timestamp_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.617726872e+09 ># HELP nginx_ingress_controller_leader_election_status Gauge reporting status of the leader election, 0 indicates follower, 1 indicates leader. 'name' is the string used to identify the lease ># TYPE nginx_ingress_controller_leader_election_status gauge >nginx_ingress_controller_leader_election_status{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",name="ingress-controller-leader-nginx"} 1 ># HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting} ># TYPE nginx_ingress_controller_nginx_process_connections gauge >nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="active"} 1 >nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="reading"} 0 >nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="waiting"} 0 >nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="writing"} 1 ># HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled} ># TYPE nginx_ingress_controller_nginx_process_connections_total counter >nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="accepted"} 713 >nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc",state="handled"} 713 ># HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds ># TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter >nginx_ingress_controller_nginx_process_cpu_seconds_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.7900000000000005 ># HELP nginx_ingress_controller_nginx_process_num_procs number of processes ># TYPE nginx_ingress_controller_nginx_process_num_procs gauge >nginx_ingress_controller_nginx_process_num_procs{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 9 ># HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01 ># TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge >nginx_ingress_controller_nginx_process_oldest_start_time_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1.61772687e+09 ># HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read ># TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter >nginx_ingress_controller_nginx_process_read_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 0 ># HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests ># TYPE nginx_ingress_controller_nginx_process_requests_total counter >nginx_ingress_controller_nginx_process_requests_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 739 ># HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use ># TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge >nginx_ingress_controller_nginx_process_resident_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.25009664e+08 ># HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use ># TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge >nginx_ingress_controller_nginx_process_virtual_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 2.155986944e+09 ># HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written ># TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter >nginx_ingress_controller_nginx_process_write_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 4096 ># HELP nginx_ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire.\n An example to check if this certificate will expire in 10 days is: "nginx_ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))" ># TYPE nginx_ingress_controller_ssl_expire_time_seconds gauge >nginx_ingress_controller_ssl_expire_time_seconds{class="nginx",host="_",namespace="ingress-nginx"} 1.649262871e+09 ># HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations ># TYPE nginx_ingress_controller_success counter >nginx_ingress_controller_success{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6dc9778bf7-phxdc"} 1 ># HELP process_cpu_seconds_total Total user and system CPU time spent in seconds. ># TYPE process_cpu_seconds_total counter >process_cpu_seconds_total 6.51 ># HELP process_max_fds Maximum number of open file descriptors. ># TYPE process_max_fds gauge >process_max_fds 1.048576e+06 ># HELP process_open_fds Number of open file descriptors. ># TYPE process_open_fds gauge >process_open_fds 39 ># HELP process_resident_memory_bytes Resident memory size in bytes. ># TYPE process_resident_memory_bytes gauge >process_resident_memory_bytes 6.0248064e+07 ># HELP process_start_time_seconds Start time of the process since unix epoch in seconds. ># TYPE process_start_time_seconds gauge >process_start_time_seconds 1.61772687031e+09 ># HELP process_virtual_memory_bytes Virtual memory size in bytes. ># TYPE process_virtual_memory_bytes gauge >process_virtual_memory_bytes 7.61356288e+08 ># HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes. ># TYPE process_virtual_memory_max_bytes gauge >process_virtual_memory_max_bytes -1 ># HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served. ># TYPE promhttp_metric_handler_requests_in_flight gauge >promhttp_metric_handler_requests_in_flight 1 ># HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code. ># TYPE promhttp_metric_handler_requests_total counter >promhttp_metric_handler_requests_total{code="200"} 110 >promhttp_metric_handler_requests_total{code="500"} 0 >promhttp_metric_handler_requests_total{code="503"} 0 >``` >So, most of dashboard are empty. >Screen Shot 2021-04-07 at 2 34 44 AM > >What am I missing? > >please helpme > >/triage support > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
KozyarValeriy commented 3 years ago

I'd had the same problem until I added the ingress resource (but I have baremetal). But now I have the same problem with TCP. TCP services don't provide metrics like nginx_ingress_controller_requests etc. Do you know what to do with that?

longwuyuan commented 3 years ago

What happens if you try this https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/

/remove-kind support /triage needs-information

KozyarValeriy commented 3 years ago

I use it. But the most part of panels with metrics are not available (No data). I can see only Average Memory Usage, Average CPU Usage, Controller Connections and Config Reloads.

longwuyuan commented 3 years ago

Show screenshots.

You won't get TCP metrics i am almost sure but not 100%

nwaistd commented 3 years ago

@longwuyuan could you give me some links to relevant docs?

KozyarValeriy commented 3 years ago

There is an nginx_ingress_controller_requests metric here, but it is not updated, because I do not use ingress resource. This appeared after the launch of ingress resource.

Prometheus metrics: all_metrics_ingress

Dashboards: dash_1_ing dash_2_ing

longwuyuan commented 3 years ago

@longwuyuan could you give me some links to relevant docs?

The link is already in the previous message. If you can install as per those docs and deploy monitoring as per those docs, maybe we will get some helpful info. But don't expect TCP traffic graphs in the dashboard you get from docs based install.

Who is your LoadBalancer provider ? Do they have monitoring for the LB they provide ?

KozyarValeriy commented 3 years ago

We use metallb (https://metallb.universe.tf/configuration/) in layer2 mode. But it doesn't provide any useful metrics.

longwuyuan commented 3 years ago

@nwaistd did you install as per the link I sent ?

@KozyarValeriy I don't think TCP metrics are exported but someone has to check and confirm. It will take a while. Hope I can check and update but very soon

nwaistd commented 3 years ago

@longwuyuan Oh, I mean about TCP metrics, why can't get TCP metrics and how to get them.

I set up ingress-controller(nlb) and prometheus in AWS EKS cluster. (k8s v1.19) as guided in the link. (added port 10254 in ingress-controller service and deployment)

NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                                          PORT(S)                                      AGE
ingress-nginx-controller             LoadBalancer   10.100.8.94     ...   80:31999/TCP,443:32245/TCP,10254:32351/TCP   40m
ingress-nginx-controller-admission   ClusterIP      10.100.104.162   <none>                                                                               443/TCP                                      40m
prometheus-server                    NodePort       10.100.227.128   <none>

                                                                               9090:32371/TCP                               37m

As you can see, all target labels are set to dropped.

Screen Shot 2021-04-24 at 2 49 56 PM

I can get following metrics including 404 status metrics in a pod, though.

 bash-5.1$ curl localhost:10254/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.198e-05
go_gc_duration_seconds{quantile="0.25"} 4.8626e-05
go_gc_duration_seconds{quantile="0.5"} 6.0267e-05
go_gc_duration_seconds{quantile="0.75"} 7.6392e-05
go_gc_duration_seconds{quantile="1"} 0.000169321
go_gc_duration_seconds_sum 0.001709751
go_gc_duration_seconds_count 26
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 90
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15.6"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 7.54696e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 8.9655112e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.476707e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.023388e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 9.220351350273963e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.632656e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 7.54696e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.5861248e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.216e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 72698
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.5631872e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.5077248e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6192428766532521e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.096086e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 138720
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 180224
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.877912e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 674061
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.998848e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.998848e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.5056128e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP nginx_ingress_controller_bytes_sent The number of bytes sent to a client
# TYPE nginx_ingress_controller_bytes_sent histogram
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100"} 0
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100000"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1e+06"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1e+07"} 1
nginx_ingress_controller_bytes_sent_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_bytes_sent_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 274
nginx_ingress_controller_bytes_sent_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_config_hash Running configuration hash actually running
# TYPE nginx_ingress_controller_config_hash gauge
nginx_ingress_controller_config_hash{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 9.155461334347864e+18
# HELP nginx_ingress_controller_config_last_reload_successful Whether the last configuration reload attempt was successful
# TYPE nginx_ingress_controller_config_last_reload_successful gauge
nginx_ingress_controller_config_last_reload_successful{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1
# HELP nginx_ingress_controller_config_last_reload_successful_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE nginx_ingress_controller_config_last_reload_successful_timestamp_seconds gauge
nginx_ingress_controller_config_last_reload_successful_timestamp_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.619240952e+09
# HELP nginx_ingress_controller_ingress_upstream_latency_seconds Upstream service latency per Ingress
# TYPE nginx_ingress_controller_ingress_upstream_latency_seconds summary
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.5"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.9"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",quantile="0.99"} NaN
nginx_ingress_controller_ingress_upstream_latency_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service=""} 0
nginx_ingress_controller_ingress_upstream_latency_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service=""} 1
# HELP nginx_ingress_controller_leader_election_status Gauge reporting status of the leader election, 0 indicates follower, 1 indicates leader. 'name' is the string used to identify the lease
# TYPE nginx_ingress_controller_leader_election_status gauge
nginx_ingress_controller_leader_election_status{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",name="ingress-controller-leader-nginx"} 1
# HELP nginx_ingress_controller_nginx_process_connections current number of client connections with state {active, reading, writing, waiting}
# TYPE nginx_ingress_controller_nginx_process_connections gauge
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="active"} 1
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="reading"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="waiting"} 0
nginx_ingress_controller_nginx_process_connections{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="writing"} 1
# HELP nginx_ingress_controller_nginx_process_connections_total total number of connections with state {accepted, handled}
# TYPE nginx_ingress_controller_nginx_process_connections_total counter
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="accepted"} 426
nginx_ingress_controller_nginx_process_connections_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",state="handled"} 426
# HELP nginx_ingress_controller_nginx_process_cpu_seconds_total Cpu usage in seconds
# TYPE nginx_ingress_controller_nginx_process_cpu_seconds_total counter
nginx_ingress_controller_nginx_process_cpu_seconds_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 0.9600000000000001
# HELP nginx_ingress_controller_nginx_process_num_procs number of processes
# TYPE nginx_ingress_controller_nginx_process_num_procs gauge
nginx_ingress_controller_nginx_process_num_procs{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 8
# HELP nginx_ingress_controller_nginx_process_oldest_start_time_seconds start time in seconds since 1970/01/01
# TYPE nginx_ingress_controller_nginx_process_oldest_start_time_seconds gauge
nginx_ingress_controller_nginx_process_oldest_start_time_seconds{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.61924095e+09
# HELP nginx_ingress_controller_nginx_process_read_bytes_total number of bytes read
# TYPE nginx_ingress_controller_nginx_process_read_bytes_total counter
nginx_ingress_controller_nginx_process_read_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 0
# HELP nginx_ingress_controller_nginx_process_requests_total total number of client requests
# TYPE nginx_ingress_controller_nginx_process_requests_total counter
nginx_ingress_controller_nginx_process_requests_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 412
# HELP nginx_ingress_controller_nginx_process_resident_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_resident_memory_bytes gauge
nginx_ingress_controller_nginx_process_resident_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 2.01281536e+08
# HELP nginx_ingress_controller_nginx_process_virtual_memory_bytes number of bytes of memory in use
# TYPE nginx_ingress_controller_nginx_process_virtual_memory_bytes gauge
nginx_ingress_controller_nginx_process_virtual_memory_bytes{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1.921888256e+09
# HELP nginx_ingress_controller_nginx_process_write_bytes_total number of bytes written
# TYPE nginx_ingress_controller_nginx_process_write_bytes_total counter
nginx_ingress_controller_nginx_process_write_bytes_total{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 4096
# HELP nginx_ingress_controller_request_duration_seconds The request processing time in milliseconds
# TYPE nginx_ingress_controller_request_duration_seconds histogram
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 1
nginx_ingress_controller_request_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_request_duration_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 0
nginx_ingress_controller_request_duration_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_request_size The request length (including request line, header, and request body)
# TYPE nginx_ingress_controller_request_size histogram
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="20"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="30"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="40"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="50"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="60"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="70"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="80"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="90"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="100"} 0
nginx_ingress_controller_request_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_request_size_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 123
nginx_ingress_controller_request_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_requests The total number of client requests.
# TYPE nginx_ingress_controller_requests counter
nginx_ingress_controller_requests{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",ingress="",namespace="",service="",status="404"} 1
# HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
# TYPE nginx_ingress_controller_response_duration_seconds histogram
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 1
nginx_ingress_controller_response_duration_seconds_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_response_duration_seconds_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 0
nginx_ingress_controller_response_duration_seconds_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_response_size The response length (including request line, header, and request body)
# TYPE nginx_ingress_controller_response_size histogram
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.005"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.01"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.025"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.05"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.25"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="0.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="1"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="2.5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="5"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="10"} 0
nginx_ingress_controller_response_size_bucket{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404",le="+Inf"} 1
nginx_ingress_controller_response_size_sum{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 274
nginx_ingress_controller_response_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd",host="_",ingress="",method="GET",namespace="",path="",service="",status="404"} 1
# HELP nginx_ingress_controller_ssl_expire_time_seconds Number of seconds since 1970 to the SSL Certificate expire.\n           An example to check if this certificate will expire in 10 days is: "nginx_ingress_controller_ssl_expire_time_seconds < (time() + (10 * 24 * 3600))"
# TYPE nginx_ingress_controller_ssl_expire_time_seconds gauge
nginx_ingress_controller_ssl_expire_time_seconds{class="nginx",host="_",namespace="ingress-nginx"} 1.650776951e+09
# HELP nginx_ingress_controller_success Cumulative number of Ingress controller reload operations
# TYPE nginx_ingress_controller_success counter
nginx_ingress_controller_success{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-6f5454cbfb-vc6hd"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.14
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 37
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.9657472e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61924095054e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.58632448e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 0
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

And this is controller log.

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v0.45.0
  Build:         7365e9eeb2f4961ef94e4ce5eb2b6e1bdb55ce5c
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.6

-------------------------------------------------------------------------------

I0424 05:47:31.877071       6 flags.go:208] "Watching for Ingress" class="nginx"
W0424 05:47:31.877142       6 flags.go:213] Ingresses with an empty class will also be processed by this Ingress controller
W0424 05:47:31.877497       6 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0424 05:47:31.877646       6 main.go:241] "Creating API client" host="https://10.100.0.1:443"
I0424 05:47:31.884117       6 main.go:285] "Running in Kubernetes cluster" major="1" minor="19+" git="v1.19.8-eks-96780e" state="clean" commit="96780e1b30acbf0a52c38b6030d7853e575bcdf3" platform="linux/amd64"
I0424 05:47:32.244619       6 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0424 05:47:32.246124       6 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
W0424 05:47:32.251336       6 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0424 05:47:32.269713       6 ssl.go:532] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0424 05:47:32.294559       6 nginx.go:254] "Starting NGINX Ingress controller"
I0424 05:47:32.302767       6 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"aff039c8-6b4f-4d38-a4c3-ab40a79a7715", APIVersion:"v1", ResourceVersion:"953557", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0424 05:47:33.494923       6 nginx.go:296] "Starting NGINX process"
I0424 05:47:33.494968       6 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0424 05:47:33.495552       6 nginx.go:316] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0424 05:47:33.495729       6 controller.go:146] "Configuration changes detected, backend reload required"
I0424 05:47:33.501761       6 status.go:84] "New leader elected" identity="ingress-nginx-controller-6f5454cbfb-vc6hd"
I0424 05:47:33.569209       6 controller.go:163] "Backend successfully reloaded"
I0424 05:47:33.569280       6 controller.go:174] "Initial sync, sleeping for 1 second"
I0424 05:47:33.569334       6 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5df4fbf9df-l8zst", UID:"8fd187a8-0149-4eed-89f5-1bdf89ab6075", APIVersion:"v1", ResourceVersion:"962870", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0424 05:48:34.984857       6 leaderelection.go:253] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0424 05:48:34.984875       6 status.go:84] "New leader elected" identity="ingress-nginx-controller-5df4fbf9df-l8zst"
longwuyuan commented 3 years ago

@nwaistd did you install as per https://kubernetes.github.io/ingress-nginx/user-guide/monitoring/ ?

nwaistd commented 3 years ago

@longwuyuan yes

longwuyuan commented 3 years ago

@nwaistd so you installed as per docs and not only TCP metrics but other metrics that are supposed to be visualized in grafana, out of the box, are also not working ?

nwaistd commented 3 years ago

@longwuyuan yes

I installed with just these three commands.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.45.0/deploy/static/provider/aws/deploy.yaml
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/prometheus/
kubectl apply --kustomize github.com/kubernetes/ingress-nginx/deploy/grafana/

All targets are dropped. So can't import sources from prometheus

Screen Shot 2021-04-26 at 3 09 15 PM
longwuyuan commented 3 years ago

Can you come on slack and discuss this ?

nwaistd commented 3 years ago

@longwuyuan I opened a thread https://kubernetes.slack.com/archives/CANQGM8BA/p1619532483367100

pchang388 commented 3 years ago

Hi @longwuyuan,

I am also experiencing the same issue. I checked the slack thread but there was no progress on it.

I am using our own Prometheus instance and have added the additional scrape config (found here: https://github.com/kubernetes/ingress-nginx/blob/master/deploy/prometheus/prometheus.yaml) to prom and also set the pod scrape annotations for the controller pod(s). I am still only gathering a subset of the metrics that the provided grafana dashboard is showing:

curl -X GET 10.1.2.127:10254/metrics | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12911    0 12911    0     0   504k      0 --:--:-- --:--:-- --:--:--  504k

171

I am using the v0.47.0 version of the controller and v2.27.1 Prometheus server. The Nginx Controller was deployed using a helm chart. Any idea on how to get the additional metrics like upstream requests/responses?

pchang388 commented 3 years ago

Hi @nwaistd,

I did some digging and found a solution that works for us, I don't have a full understanding of everything so this may not be applicable for you or 100% correct but it seems the missing metrics are no longer present due to the commit made in the past as outlined in this issue: https://github.com/kubernetes/ingress-nginx/issues/3713. This commit made it so that ingress objects that do not specify a value in spec.rules.host field (which we are not doing currently in our development phase) will not have metrics counted in order to address valid requests vs DOS requests.

A workaround was implemented for this here in this pull request: https://github.com/kubernetes/ingress-nginx/pull/4139 and can be used by specifying --metrics-per-host=false in the args when launching the controller.

nwaistd commented 3 years ago

@pchang388 Thank you so much! I'll test it and leave a comment soon.

nwaistd commented 3 years ago

I added host in ingress, and it worked.

...
spec:
  rules:
  - host: "<MY_HOST_NAME>"
     http:
       paths:
         ...

I can get various status metrics from localhost:10254/metrics like this.

...
nginx_ingress_controller_response_size_count{controller_class="nginx",controller_namespace="ingress-nginx",controller_pod="ingress-nginx-controller-78755949f6-mpn4t",host="<MY_HOST_NAME>",ingress="xxxxxx",method="POST",namespace="default",path="xxxxxx",service="xxxxxx",status="200"} 2
...
k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/7025#issuecomment-976358845): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.