Open bismarck opened 3 years ago
I downgraded M3 query to 0.15.17 (matching the M3DB version) and the M3-Limit-Require-Exhaustive
works correctly.
We also did some debugging with @robskillington and it looks like we're hitting the M3-Limit-Max-Series
at the node while scanning the 6h block (correct me if I'm wrong). May want to update the docs about where this limit can be hit?
Description
When
M3-Limit-Max-Series
header is set, some queries are returningM3-Results-Limited: max_fetch_series_limit_applied
when they shouldn't have hit the series limit. While testing this I discovered that settingM3-Limit-Require-Exhaustive
to true does not return an error when going over the series limit.Examples
M3-Limit-Max-Series
The query
count(container_start_time_seconds)
should touch 769 time series.M3-Limit-Require-Exhaustive
Config
M3 Query
Version:
coordinator_build_information{branch="HEAD",build_date="2020_11_17_23_23_36",build_version="v1_0_0",go_version="go1_13_15",revision="a3853ee56"}
Config:M3DB
Version:
build_information{branch="HEAD",build_date="2020_10_28_19_57_32",build_version="v0_15_17_hotfix_2",go_version="go1_13_8",revision="313c1e482"}
Config:Additional Info
I ran
sum by (result, exhaustive) (rate(dbindex_query[1m]))
and all the*_require_exhaustive
results were 0.I also tried replicating the issue with the prometheus docker integration tests in the M3 repo. I modified the test suite to scrape an avalanche instance that was producing a single metric with 10,000 series. Unfortunately I wasn't able to reproduce the issue we're seeing in our production cluster.