Open bobrik opened 9 years ago
manolama
commented
9 years ago Hmm, I'd suspect that it's something in the aggregation that's throwing it off. Can you find out what's in the
"aggregateTags": [
"more",
"tags"
],by isolating down to a single series, then try the 1m, 10m etc on it?
bobrik
commented
9 years ago Isolated to empty aggregateTags:
[
{
"metric": "kafka.topics.offsets",
"tags": {
"some": "tags"
},
"aggregateTags": [],
"dps": {
"1443532260": 69905.06666666667,
"1443532380": 72089.6,
"1443532500": 74274.13333333333,
"1443532620": 85196.8,
"1443532740": 83012.26666666666,
"1443532860": 80827.73333333334,
"1443533010": 61166.933333333334,
"1443533130": 78643.2,
"1443533250": 78643.2,
"1443533370": 78643.2,
"1443533490": 80827.73333333334,
"1443533610": 78643.2,
"1443533730": 78643.2,
"1443533850": 78643.2,
"1443533970": 78643.2,
"1443534090": 76458.66666666667,
"1443534210": 78643.2,
"1443534360": 62914.56,
"1443534480": 78643.2,
"1443534600": 78643.2,
"1443534720": 78643.2,
"1443534840": 78643.2,
"1443534960": 76458.66666666667,
"1443535080": 83012.26666666666,
"1443535200": 76458.66666666667,
"1443535320": 80827.73333333334,
"1443535470": 62914.56,
"1443535590": 78643.2,
"1443535710": 74274.13333333333,
"1443535830": 74274.13333333333,
"1443535950": 72089.6,
"1443536070": 74274.13333333333,
"1443536190": 78643.2,
"1443536310": 76458.66666666667,
"1443536430": 74274.13333333333,
"1443536550": 76458.66666666667,
"1443536700": 59419.306666666664,
"1443536820": 76458.66666666667,
"1443536940": 74274.13333333333,
"1443537060": 76458.66666666667,
"1443537180": 76458.66666666667,
"1443537300": 78643.2,
"1443537420": 76458.66666666667,
"1443537570": 64662.18666666667,
"1443537690": 76458.66666666667,
"1443537810": 76458.66666666667,
"1443537930": 76458.66666666667,
"1443538050": 76458.66666666667,
"1443538170": 76458.66666666667,
"1443538290": 74274.13333333333,
"1443538410": 76458.66666666667,
"1443538530": 74274.13333333333,
"1443538650": 76458.66666666667,
"1443538770": 76458.66666666667,
"1443538890": 76458.66666666667,
"1443539040": 62914.56,
"1443539160": 74274.13333333333,
"1443539280": 74274.13333333333,
"1443539400": 74274.13333333333,
"1443539520": 74274.13333333333,
"1443539640": 76458.66666666667,
"1443539760": 74274.13333333333,
"1443539880": 76458.66666666667,
"1443540000": 74274.13333333333,
"1443540120": 74274.13333333333,
"1443540270": 61166.933333333334,
"1443540390": 74274.13333333333,
"1443540510": 74274.13333333333,
"1443540630": 76458.66666666667,
"1443540750": 78643.2,
"1443540870": 74274.13333333333,
"1443540990": 76458.66666666667,
"1443541110": 76458.66666666667,
"1443541230": 76458.66666666667,
"1443541350": 76458.66666666667,
"1443541470": 76458.66666666667,
"1443541620": 61166.933333333334,
"1443541740": 76458.66666666667,
"1443541860": 74274.13333333333,
"1443541980": 76458.66666666667,
"1443542100": 76458.66666666667,
"1443542220": 76458.66666666667,
"1443542340": 78643.2,
"1443542460": 78643.2,
"1443542580": 80827.73333333334,
"1443542700": 80827.73333333334,
"1443542820": 80827.73333333334,
"1443542970": 62914.56,
"1443543090": 80827.73333333334,
"1443543210": 78643.2,
"1443543330": 80827.73333333334,
"1443543450": 80827.73333333334,
"1443543570": 78643.2,
"1443543690": 80827.73333333334,
"1443543810": 78643.2,
"1443543930": 80827.73333333334,
"1443544050": 78643.2,
"1443544200": 64662.18666666667,
"1443544320": 78643.2,
"1443544440": 76458.66666666667,
"1443544560": 80827.73333333334,
"1443544800": 78643.2,
"1443544920": 78643.2,
"1443545040": 78643.2,
"1443545160": 76458.66666666667,
"1443545280": 78643.2,
"1443545400": 78643.2,
"1443545550": 62914.56,
"1443545670": 78643.2,
"1443545790": 76458.66666666667,
"1443545910": 78643.2,
"1443546030": 78643.2,
"1443546150": 76458.66666666667,
"1443546270": 78643.2,
"1443546390": 74274.13333333333,
"1443546510": 74274.13333333333,
"1443546630": 74274.13333333333,
"1443546750": 74274.13333333333,
"1443546900": 57671.68,
"1443547020": 74274.13333333333,
"1443547140": 74274.13333333333,
"1443547260": 72089.6,
"1443547380": 74274.13333333333,
"1443547500": 72089.6,
"1443547620": 72089.6,
"1443547740": 72089.6,
"1443547860": 72089.6,
"1443547980": 74274.13333333333,
"1443548130": 57671.68,
"1443548250": 72089.6,
"1443548370": 72089.6,
"1443548490": 69905.06666666667,
"1443548610": 72089.6,
"1443548730": 72089.6,
"1443548850": 72089.6,
"1443548970": 72089.6,
"1443549090": 72089.6,
"1443549210": 72089.6,
"1443549330": 72089.6,
"1443549480": 57671.68,
"1443549600": 72089.6,
"1443549840": 72089.6,
"1443549960": 69905.06666666667,
"1443550080": 72089.6,
"1443550200": 72089.6,
"1443550440": 70997.33333333333,
"1443550680": 70997.33333333333,
"1443550950": 64079.64444444444,
"1443551070": 72089.6,
"1443551430": 72089.6,
"1443551550": 72089.6,
"1443551670": 72089.6,
"1443551790": 74274.13333333333,
"1443551910": 72089.6,
"1443552030": 72089.6,
"1443552180": 57671.68,
"1443552300": 72089.6,
"1443552420": 72089.6,
"1443552540": 74274.13333333333,
"1443552660": 72089.6,
"1443552780": 74274.13333333333,
"1443552900": 72089.6,
"1443553020": 72089.6,
"1443553140": 72089.6,
"1443553260": 72089.6,
"1443553380": 69905.06666666667,
"1443553530": 55924.05333333334,
"1443553650": 72089.6,
"1443553770": 72089.6
}
}
]Issue is still there with 1m, but not with bigger intervals:
manolama
commented
9 years ago OK, thanks, let me play around with that data set to see what's going on.
bobrik
commented
9 years ago Looks like I gave you downsampled result, here's the result for disabled downsampling:
[
{
"metric": "kafka.topics.offsets",
"tags": {
"some": "tags"
},
"aggregateTags": [],
"dps": {
"1443533381": 77353.96721311475,
"1443533504": 78856.32520325204,
"1443533626": 77353.96721311475,
"1443533749": 76725.0731707317,
"1443533871": 77353.96721311475,
"1443533994": 76725.0731707317,
"1443534117": 74593.82113821138,
"1443534239": 77353.96721311475,
"1443534362": 76725.0731707317,
"1443534484": 77353.96721311475,
"1443534607": 76725.0731707317,
"1443534729": 77353.96721311475,
"1443534852": 76725.0731707317,
"1443534976": 73992.25806451614,
"1443535102": 79059.30158730158,
"1443535225": 74593.82113821138,
"1443535347": 79502.68852459016,
"1443535470": 76725.0731707317,
"1443535593": 76725.0731707317,
"1443535716": 72462.56910569106,
"1443535838": 73056.52459016393,
"1443535962": 69764.12903225806,
"1443536085": 72462.56910569106,
"1443536210": 75497.472,
"1443536333": 74593.82113821138,
"1443536455": 73056.52459016393,
"1443536578": 74593.82113821138,
"1443536702": 71878.19354838709,
"1443536824": 75205.24590163934,
"1443536946": 73056.52459016393,
"1443537070": 73992.25806451614,
"1443537192": 75205.24590163934,
"1443537318": 74898.28571428571,
"1443537444": 72817.77777777778,
"1443537570": 76978.79365079365,
"1443537692": 75205.24590163934,
"1443537817": 73400.32,
"1443537939": 75205.24590163934,
"1443538061": 75205.24590163934,
"1443538184": 74593.82113821138,
"1443538306": 73056.52459016393,
"1443538429": 74593.82113821138,
"1443538552": 72462.56910569106,
"1443538674": 75205.24590163934,
"1443538797": 74593.82113821138,
"1443538919": 75205.24590163934,
"1443539042": 76725.0731707317,
"1443539165": 72462.56910569106,
"1443539289": 71878.19354838709,
"1443539412": 72462.56910569106,
"1443539535": 72462.56910569106,
"1443539657": 75205.24590163934,
"1443539780": 72462.56910569106,
"1443539902": 75205.24590163934,
"1443540026": 71878.19354838709,
"1443540148": 73056.52459016393,
"1443540270": 75205.24590163934,
"1443540392": 73056.52459016393,
"1443540515": 72462.56910569106,
"1443540638": 74593.82113821138,
"1443540761": 76725.0731707317,
"1443540884": 72462.56910569106,
"1443541006": 75205.24590163934,
"1443541129": 74593.82113821138,
"1443541251": 75205.24590163934,
"1443541375": 73992.25806451614,
"1443541498": 74593.82113821138,
"1443541621": 74593.82113821138,
"1443541744": 74593.82113821138,
"1443541866": 73056.52459016393,
"1443541989": 74593.82113821138,
"1443542111": 75205.24590163934,
"1443542235": 73992.25806451614,
"1443542357": 77353.96721311475,
"1443542481": 76106.32258064517,
"1443542604": 78856.32520325204,
"1443542726": 79502.68852459016,
"1443542849": 78856.32520325204,
"1443542972": 76725.0731707317,
"1443543094": 79502.68852459016,
"1443543218": 76106.32258064517,
"1443543340": 79502.68852459016,
"1443543464": 78220.3870967742,
"1443543586": 77353.96721311475,
"1443543708": 79502.68852459016,
"1443543832": 76106.32258064517,
"1443543955": 78856.32520325204,
"1443544079": 76106.32258064517,
"1443544202": 78856.32520325204,
"1443544324": 77353.96721311475,
"1443544447": 74593.82113821138,
"1443544569": 79502.68852459016,
"1443544814": 77038.23673469388,
"1443544937": 76725.0731707317,
"1443545060": 76725.0731707317,
"1443545183": 74593.82113821138,
"1443545306": 76725.0731707317,
"1443545429": 76725.0731707317,
"1443545551": 77353.96721311475,
"1443545674": 76725.0731707317,
"1443545797": 74593.82113821138,
"1443545919": 77353.96721311475,
"1443546042": 76725.0731707317,
"1443546164": 75205.24590163934,
"1443546288": 76106.32258064517,
"1443546411": 72462.56910569106,
"1443546533": 73056.52459016393,
"1443546656": 72462.56910569106,
"1443546778": 73056.52459016393,
"1443546901": 70331.31707317074,
"1443547025": 71878.19354838709,
"1443547147": 73056.52459016393,
"1443547269": 70907.80327868853,
"1443547392": 72462.56910569106,
"1443547514": 70907.80327868853,
"1443547637": 70331.31707317074,
"1443547759": 70907.80327868853,
"1443547884": 69206.016,
"1443548006": 73056.52459016393,
"1443548130": 69764.12903225806,
"1443548253": 70331.31707317074,
"1443548375": 70907.80327868853,
"1443548497": 68759.08196721312,
"1443548620": 70331.31707317074,
"1443548743": 70331.31707317074,
"1443548866": 70331.31707317074,
"1443548989": 70331.31707317074,
"1443549111": 70907.80327868853,
"1443549234": 70331.31707317074,
"1443549356": 70907.80327868853,
"1443549482": 68656.76190476191,
"1443549604": 70907.80327868853,
"1443549850": 70331.31707317074,
"1443549972": 68759.08196721312,
"1443550095": 70331.31707317074,
"1443550218": 70331.31707317074,
"1443550463": 69548.40816326531,
"1443550708": 69548.40816326531,
"1443550954": 70331.31707317074,
"1443551077": 70331.31707317074,
"1443551446": 70331.31707317074,
"1443551568": 70907.80327868853,
"1443551691": 70331.31707317074,
"1443551813": 73056.52459016393,
"1443551936": 70331.31707317074,
"1443552059": 70331.31707317074,
"1443552181": 70907.80327868853,
"1443552303": 70907.80327868853,
"1443552425": 70907.80327868853,
"1443552548": 72462.56910569106,
"1443552672": 69764.12903225806,
"1443552794": 73056.52459016393,
"1443552917": 70331.31707317074,
"1443553039": 70907.80327868853,
"1443553162": 70331.31707317074,
"1443553286": 69764.12903225806,
"1443553409": 68200.06504065041,
"1443553531": 68759.08196721312,
"1443553654": 70331.31707317074,
"1443553777": 70331.31707317074,
"1443553899": 68759.08196721312,
"1443554023": 67650.06451612903,
"1443554145": 70907.80327868853,
"1443554267": 70907.80327868853,
"1443554392": 69206.016,
"1443554514": 70907.80327868853,
"1443554637": 70331.31707317074,
"1443554759": 68759.08196721312,
"1443554882": 68200.06504065041
}
}
]I think I've found the issue. Imagine rate 1/s with the following datapoints and 1m-avg downsampler:
12:00:01 value=1 downsamples to 12:00:00 value=112:01:59 value=119 downsamples to 12:01:00 value=119Calculated rate:
(119 - 1) / (12:01:00 - 12:00:00) = 118 / 60 = 1.96We get 1.96 while the real value is 1.0. The reason for this is the way downsampler works: it fills the whole interval and runs aggregator on datapoints in this interval.
bobrik
commented
8 years ago It seems like interpolated values should be calculated at the edges of downsampling intervals before running aggregations to have nice results. In my example, the following buckets should be used:
12:00:00, values in between (12:00:01), 12:00:59 — for 12:00:00 bucket.12:01:00, values in between (12:01:59), 12:01:59 — for 12:00:01 bucket.
xblitz
commented
7 years ago any news on this? I seem to have the same issues. my metric is a every growing packet loss count, so I use the rate function, but if I want to downsample (with a sum for example) well I get weird spikes
here is the raw datapoints: (we can observe a late polling at midnight
but when downsampled it creates a spike because of how it is handled
so if I try to use the rate option (with downsample) it gives me false readings:
Tried to search for similar issue, but haven't found one. I'm running 2.1.0, every graph below is taken from 6h timespan.
Query looks roughly like this:
Query result for disabled downsampling: