Filtering metrics by labelSelector in external.metrics.k8s.io api doesn't work

[5imun]

Some metrics labels (like resource.labels.instance_id in my case) are longer than 63 characters and have forbidden characters like slashes which makes filtering by those labels impossible and thus whole v1beta1.external.metrics.k8s.io service unusable for big number of external metrics.

If I run this with simpler labels like resource.labels.region I will get external metrics for all redis instances in the region as expected:

kubectl get --raw  "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/redis.googleapis.com|stats|network_traffic?labelSelector=resource.labels.region=europe-west3" | jq .

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "redis.googleapis.com|stats|network_traffic",
      "metricLabels": {
        "metric.labels.direction": "in",
        "metric.labels.role": "primary",
        "resource.labels.instance_id": "projects/test-project/locations/europe-west3/instances/production",
        "resource.labels.node_id": "node-0",
        "resource.labels.project_id": "test-project",
        "resource.labels.region": "europe-west3",
        "resource.type": "redis_instance"
      },
      "timestamp": "2023-04-19T09:28:42Z",
      "value": "3945867400m"
    },
    ...
    {
      "metricName": "redis.googleapis.com|stats|network_traffic",
      "metricLabels": {
        "metric.labels.direction": "in",
        "metric.labels.role": "primary",
        "resource.labels.instance_id": "projects/test-project/locations/europe-west3/instances/staging",
        "resource.labels.node_id": "node-0",
        "resource.labels.project_id": "test-project",
        "resource.labels.region": "europe-west3",
        "resource.type": "redis_instance"
      },
      "timestamp": "2023-04-19T09:28:42Z",
      "value": "13748666m"
    },
  ]
}

Problem is if I want to select metric for specific redis instance using resource.labels.instance_id label that is not in allowed format then I get error:

kubectl get --raw  "/apis/external.metrics.k8s.io/v1beta1/namespaces/*/redis.googleapis.com|stats|network_traffic?labelSelector=resource.labels.instance_id=projects/test-project/locations/europe-west3/instances/staging" | jq .

Error from server (BadRequest): unable to parse requirement: values[0][resource.labels.instance_id]: Invalid value: "projects/test-project/locations/europe-west3/instances/staging": must be no more than 63 characters; a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

This makes horizontal pod autoscaling based on external metric impossible if you want to scale pods based on metrics from specific redis instance in the google cloud project.

[CatherineF-dev]

have forbidden characters like slashes

Similar to metric names, needs to replace slashes with | to make it work. This is not implemented yet. Contributions to this repo are welcomed.

are longer than 63 characters

Haven't found an existing solution yet. I think one possibility is to use GMP front-end + prometheus adapter.