search

jacksontj / promxy

An aggregating proxy to enable HA prometheus
MIT License
1.12k stars 126 forks source link

The count function does not execute correctly #632

Closed resurgence72 closed 7 months ago

resurgence72 commented 7 months ago

hi，At present, our tsdb is only used as a storage role, and I used three tsdb as data shard storage, that is, each tsdb only stores 1/3 of the data. And promxy is used as the query side. Currently I have a simple promql 'prober_agent_is_alive==1' that executes alone and hits 17 series； image Then I want to use the count function to calculate the total number of the series, but the final result is 8, in my understanding, count should be the same as the total number of the series. image

But when I used the prometheus remote read structure, the statement returned the correct result. image

resurgence72 commented 7 months ago

hi,In my tests, the sum function also returns 8；this is my config image

resurgence72 commented 7 months ago

After I removed the 'anti_affinity' configuration, the query returned to normal. It seems that it is caused by anti_affinity. I need to study the function of this parameter again.

jacksontj commented 6 months ago

@resurgence72 from a quick glance at this issue; I believe the issue is with your servergroup configuration. What you described above is that each of these 3 shards hold unique data; if so then they should each have their own servergroup config with labels (discussion about this in https://github.com/jacksontj/promxy/issues/260).

Otherwise promxy believes the 3 have the same data and will only "fill" if there are holes -- which there probably aren't unless there is no anti_affinity config (which forces it to merge always).

Hopefully that'll help!