search

kubernetes-sigs / prometheus-adapter

An implementation of the custom.metrics.k8s.io API using Prometheus
Apache License 2.0
1.9k stars 551 forks source link

Query too long #668

Open abursavich opened 1 month ago

abursavich commented 1 month ago

What happened?:

The query that matches all ready pods in a large deployment is being rejected by VictoriaMetrics.

E0703 20:49:59.724673 1 provider.go:150] unable to fetch metrics from prometheus: 422: too long query; got 25852 bytes; mustn't exceed -search.maxQueryLen=16384 bytes

We can (and will for now) increase the maximum size of a query that VictoriaMetrics allows, but we could also reduce the size of the query that prometheus-adapter constructs.

Given these pods:

  1. podfromareplicasetwithalongname-24vm7
  2. podfromareplicasetwithalongname-4h7zn
  3. podfromareplicasetwithalongname-gpmg2
  4. podfromareplicasetwithalongname-scgch
  5. podfromareplicasetwithalongname-x7ks6

The constructed query is: some_metric_name{namespace="foobar",pod=~"podfromareplicasetwithalongname-24vm7|podfromareplicasetwithalongname-4h7zn|podfromareplicasetwithalongname-gpmg2|podfromareplicasetwithalongname-scgch|podfromareplicasetwithalongname-x7ks6"}

But it could be: some_metric_name{namespace="foobar",pod=~""podfromareplicasetwithalongname-(24vm7|4h7zn|gpmg2|scgch|x7ks6)"}

Now imagine that with 1500 pods instead of 5.

What did you expect to happen?:

Prometheus-adapter should make a reasonable effort to create small queries.

dgrisonnet commented 1 month ago

/triage accepted /assign @abursavich