基于Custom-metrics-apiserver实现Kubernetes的HPA

部署 kube-prometheus-stack (prometheus-operator)

  additionalServiceMonitors: 
    - name: myapp-monitor
      selector:
        matchLabels:
          app.kubernetes.io/instance: myapp
          app.kubernetes.io/name: myapp
      namespaceSelector:
        # any: true
        matchNames:
          - myapp
      endpoints:
        - port: http
          interval: 10s
          path: /actuator/prometheus

部署 prometheus-adapter

rules:
  default: false

  custom: 
    - seriesQuery: 'myapp_http_requests_count'
      resources:
        template: <<.Resource>>
      name:
        matches: "^(.*)_count"
        as: "${1}_per_second"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

記得覆寫 prometheus.url 為 http://\<kube-prometheus-stack service>.\<namespace>.svc.cluster.local

透過 --set prometheus.url=

helm install <adapter-name> -n <namespace> --set prometheus.url=http://kube-prometheus-stack.monitoring.svc.cluster.local --set prometheus.port=9090 --set prometheus.path=/

透過 -f values.yaml

...
prometheus:
url: http://prometheus-opreated.monitoring.svc
port: 9090
path: /
...

部署 APP 和 Service

建立 HPA

基於 POD

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-custom-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deploy-v1.0
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: myapp_http_requests_count
      target:
        type: AverageValue
        averageValue: 5000m

自訂資源指標含義是基於 myapp-deploy-v1.0 這個 deployment 的全部 pod 來計算http_requests的平局值，如果達到 5000m就進行擴容。

scaleTargetRef：這裡定義的是自動擴容縮容的對象，可以是Deployment或者ReplicaSet，這裡寫具體的Deployment的名稱。
metrics：這裡是指標的目標值。在type中定義類型；通過target來定義指標的閾值，系統將在指標達到閾值的時候出發擴縮容操作。 metrics中的type有如下類型：
- Resource：基於資源的指標，可以是CPU或者是記憶體，如果基於這個類型的指標來做只需要部署Metric-server即可，不需要部署自訂APISERVER。
- Pods：基於Pod的指標，系統將對Deployment中的全部Pod副本指標進行平均值計算，如果是Pod則該指標必須來源於Pod本身。
- Object：基於Ingress或者其他自訂指標，比如ServiceMonitor。它的target類型可以是Value或者AverageValue（根據Pod副本數計算平均值）。

基於 Service

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-custom-metrics-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deploy-v1.0
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Object
    object:
      target:
        kind: Service
        # 這裡是你自己的APP的service
        name: myapp-svc
      metric:
        name: http_requests
      target:
        type: AverageValue
        averageValue: 200
--
apiVersion: v1
kind: Service
metadata:
  name: myapp-svc
  labels:
    appname: myapp-svc
  annotations:
    prometheus.io/scrape: "true"  # 新增內容
    prometheus.io/port: "5555"  # 新增內容
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 5555
    targetPort: 5555
  selector:
    appname: myapp

然後 prometheus-adapter 規則，增加如下內容：

rules:
  default: false

  custom:
    - seriesQuery: '{__name__=~"^http_requests_.*",kubernetes_name!="",kubernetes_namespace!=""}'
      seriesFilters: []
      resources:
        overrides:
          kubernetes_namespace:
            resource: namespace
          kubernetes_name:
            resource: service
      name:
        matches: ^(.*)_(total)$
        as: "${1}"
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)

mangreen / Some-Note

Horizontal Pod Autoscale with Custom Prometheus Metrics #26