The query that matches all ready pods in a large deployment is being rejected by VictoriaMetrics.
E0703 20:49:59.724673 1 provider.go:150] unable to fetch metrics from prometheus: 422: too long query; got 25852 bytes; mustn't exceed -search.maxQueryLen=16384 bytes
We can (and will for now) increase the maximum size of a query that VictoriaMetrics allows, but we could also reduce the size of the query that prometheus-adapter constructs.
Given these pods:
podfromareplicasetwithalongname-24vm7
podfromareplicasetwithalongname-4h7zn
podfromareplicasetwithalongname-gpmg2
podfromareplicasetwithalongname-scgch
podfromareplicasetwithalongname-x7ks6
The constructed query is:
some_metric_name{namespace="foobar",pod=~"podfromareplicasetwithalongname-24vm7|podfromareplicasetwithalongname-4h7zn|podfromareplicasetwithalongname-gpmg2|podfromareplicasetwithalongname-scgch|podfromareplicasetwithalongname-x7ks6"}
But it could be:
some_metric_name{namespace="foobar",pod=~""podfromareplicasetwithalongname-(24vm7|4h7zn|gpmg2|scgch|x7ks6)"}
Now imagine that with 1500 pods instead of 5.
What did you expect to happen?:
Prometheus-adapter should make a reasonable effort to create small queries.
What happened?:
The query that matches all ready pods in a large deployment is being rejected by VictoriaMetrics.
We can (and will for now) increase the maximum size of a query that VictoriaMetrics allows, but we could also reduce the size of the query that prometheus-adapter constructs.
Given these pods:
The constructed query is:
some_metric_name{namespace="foobar",pod=~"podfromareplicasetwithalongname-24vm7|podfromareplicasetwithalongname-4h7zn|podfromareplicasetwithalongname-gpmg2|podfromareplicasetwithalongname-scgch|podfromareplicasetwithalongname-x7ks6"}
But it could be:
some_metric_name{namespace="foobar",pod=~""podfromareplicasetwithalongname-(24vm7|4h7zn|gpmg2|scgch|x7ks6)"}
Now imagine that with 1500 pods instead of 5.
What did you expect to happen?:
Prometheus-adapter should make a reasonable effort to create small queries.