search

GoogleCloudPlatform / k8s-stackdriver

Apache License 2.0
390 stars 212 forks source link

Empty external metrics and retrieving the list under custom metrics lead to HPA not working #728

Closed mohsen-vybenetwork closed 4 months ago

mohsen-vybenetwork commented 4 months ago

Hi,

I've deployed k8s-stackdriver but it's so weird I'm getting metrics list under custom metrics and not external metrics.

kubectl get --raw '/apis/external.metrics.k8s.io' | jq
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "external.metrics.k8s.io",
  "versions": [
    {
      "groupVersion": "external.metrics.k8s.io/v1beta1",
      "version": "v1beta1"
    }
  ],
  "preferredVersion": {
    "groupVersion": "external.metrics.k8s.io/v1beta1",
    "version": "v1beta1"
  }
kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta2' | jq . |grep -i loadba
      "name": "*/loadbalancing.googleapis.com|https|backend_request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|backend_request_count",
      "name": "*/loadbalancing.googleapis.com|https|backend_response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|backend_request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|backend_request_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|backend_response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|request_count",
      "name": "*/loadbalancing.googleapis.com|https|external|regional|response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|backend_request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|backend_request_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|backend_response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|request_count",
      "name": "*/loadbalancing.googleapis.com|https|internal|response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|request_bytes_count",
      "name": "*/loadbalancing.googleapis.com|https|request_count",
      "name": "*/loadbalancing.googleapis.com|https|response_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l3|external|egress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l3|external|egress_packets_count",
      "name": "*/loadbalancing.googleapis.com|l3|external|ingress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l3|external|ingress_packets_count",
      "name": "*/loadbalancing.googleapis.com|l3|internal|egress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l3|internal|egress_packets_count",
      "name": "*/loadbalancing.googleapis.com|l3|internal|ingress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l3|internal|ingress_packets_count",
      "name": "*/loadbalancing.googleapis.com|l4_proxy|egress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l4_proxy|ingress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|l4_proxy|tcp|closed_connections_count",
      "name": "*/loadbalancing.googleapis.com|subnet|proxy_only|addresses",
      "name": "*/loadbalancing.googleapis.com|tcp_ssl_proxy|closed_connections",
      "name": "*/loadbalancing.googleapis.com|tcp_ssl_proxy|egress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|tcp_ssl_proxy|ingress_bytes_count",
      "name": "*/loadbalancing.googleapis.com|tcp_ssl_proxy|new_connections",
      "name": "*/loadbalancing.googleapis.com|tcp_ssl_proxy|open_connections",
      "name": "*/network.googleapis.com|loadbalancer|max_rate",
      "name": "*/network.googleapis.com|loadbalancer|max_utilization",
      "name": "*/network.googleapis.com|loadbalancer|utilization",

Also getting getting this error unable to get external metric api/loadbalancing.googleapis.com|https|request_count/&LabelSelector{MatchLabels:map[string]string{resource.labels.forwarding_rule_name: k8s2-fs-1ccivh5j-api-data-api-ingress-0978a4gg,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server is currently unable to handle the request (get loadbalancing.googleapis.com|https|request_count.external.metrics.k8s.io) which seems it looks for external.metrics.k8s.io and not custom.metrics.k8s.io.

And HPA is a simple hpa nothing crazy

 metrics:
  - type: External
    external:
      metric:
        name: loadbalancing.googleapis.com|https|request_count
        selector:
          matchLabels:
            resource.labels.forwarding_rule_name: k8s2-fs-1ccivh5j-api-data-api-ingress-0978a4gg
      target:
        type: AverageValue
        averageValue: 5
mohsen-vybenetwork commented 4 months ago

Fixed. issue was related to having multiple external metrics adapter.