search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

Inconsistent Downsample Output #2222

Open megha10 opened 2 years ago

megha10 commented 2 years ago

OpenTSDB read does not give a consistent output when we try with different start and end ranges and also with the RAW data points. Listed the inconsistent scenarios below,

Scenarios

  1. 0all-sum is not matching with the sum of 1d-sum for the same start and end timestamps. [ API Call - 1 and API Call - 2 ]
  2. 1d-sum returns different values if start and end timestamps are different. [ API Call - 3 and API Call - 2 ]
  3. Sum of raw data points is not matching the 1d-sum value [ API Call - 4 and API Call - 2 ]

Scenario-1)

API Call -1

curl -k --request POST \
  --url http://opentsdb-read.run:4242/api/query?details \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{
  "start": "1640995200",
  "end": "1642723199",
  "padding": false,
  "queries": [
    {
      "aggregator": "sum",
      "metric": "successful_transactions",
      "downsample": "0all-sum",
        "tags": {
          "tagA": "dummy"
        }
    }
  ]
}'

…
"dps": {
      "1640995200": 270389110471438       —----- (1)
    }
…

API Call -2

curl -k --request POST \
  --url http://opentsdb-read.run:4242/api/query?details \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{
  "start": "1640995200",
  "end": "1642723199",
  "padding": false,
  "queries": [
    {
      "aggregator": "sum",
      "metric": "successful_transactions",
      "downsample": "1d-sum",
      "rollupUsage": "ROLLUP_RAW",
      "tags": {
          "tagA": "dummy"
        }
    }
  ]
}'

"dps": {
      "1640995200": 9239688871045,
      "1641081600": 9374911382649.5,
      "1641168000": 11121302694203,
      "1641254400": 11465546332867.5,
      "1641340800": 11080449815436,
      "1641427200": 10974654781071,
      "1641513600": 14980428874584,
      "1641600000": 14272579470839.97,     —------ (3)
      "1641686400": 13900739068238.451,   —------ (4)
      "1641772800": 16478902447667.184,
      "1641859200": 15026058712648.072,
      "1641945600": 14824122916484.836,
      "1642032000": 15076325746403.7,
      "1642118400": 15057402616820.07,
      "1642204800": 14015163439695.436,
      "1642291200": 13155230552208.805,
      "1642377600": 14935039332635.504,
      "1642464000": 15370715216576.152,
      "1642550400": 15791470679695.215,
      "1642636800": 15122326168536.855
    }

Sum of the above data points is 271263059120306. This is not matching with the 0all-sum value 270389110471438.(API Call -1)

Scenario-2)

API Call - 3

curl -k --request POST \
  --url http://opentsdb-read.run:4242/api/query?details \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{
  "start": "1641600000",
  "end": "1641686400",
  "padding": false,
  "queries": [
    {
      "aggregator": "sum",
      "metric": "successful_transactions",
      "downsample": "1d-sum",
      "rollupUsage": "ROLLUP_RAW",
        "tags": {
          "tagA": "dummy"
        }
    }
  ]
}' | jq '.'

"dps": {
      "1641600000": 14037951308980,
      "1641686400": 13479957768414
    }

Above numbers are not matching with the API Call -2’s 1641600000 & 1641686400 timestamp values.

Scenario-3)

API Call - 4

curl -k --request POST \
  --url http://opentsdb-read.run:4242/api/query?details \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --data '{
  "start": "1641600000",
  "end": "1641686399",
  "padding": false,
  "queries": [
    {
      "aggregator": "sum",
      "metric": "successful_transactions",
      "downsample": "5m-sum",
      "rollupUsage": "ROLLUP_RAW",
        "tags": {
          "tagA": "dummy"
        }
    }
  ]
}' 

"dps": {
      "1641600000": 41483770657.5,
      "1641600300": 37868925499.5,
      "1641600600": 41527628066,
      "1641600900": 37174736840.5,
      "1641601200": 35819626035.5,
      "1641601500": 34571615007,
      "1641601800": 35837905461,
      "1641602100": 33960782059,
      "1641602400": 25133478459,
      "1641602700": 33134261586.5,
      "1641603000": 36213078116,
      "1641603300": 32390077481,
      "1641603600": 35308007123.5,
      "1641603900": 33567605059.5,
      "1641604200": 39721090098.5,
      "1641604500": 34447327053,
      "1641604800": 31884873261.5,
      "1641605100": 30844737683.5,
      "1641605400": 31681226716.5,
      "1641605700": 29927713173,
      "1641606000": 31326186992.5,
      "1641606300": 32726823233,
      "1641606600": 35730069122.5,
      "1641606900": 31921166151.5,
      "1641607200": 34357449226,
      "1641607500": 32614837951.5,
      "1641607800": 30305825229.5,
      "1641608100": 33629186327,
      "1641608400": 27591850173,
      "1641608700": 28500995994.52128,
      "1641609000": 29343491772.542553,
      "1641609300": 29823838841.56383,
      "1641609600": 29113358204.585106,
      "1641609900": 27897114700.606384,
      "1641610200": 28082706021.12766,
      "1641610500": 27777080785.648937,
      "1641610800": 49672153347.17021,
      "1641611100": 53510418754.69149,
      "1641611400": 41584830608.21277,
      "1641611700": 27863674440.734043,
      "1641612000": 28279389750.755318,
      "1641612300": 24360423010.276596,
      "1641612600": 27195749745.79787,
      "1641612900": 26720986192.31915,
      "1641613200": 26699347171.840427,
      "1641613500": 25834919501.861702,
      "1641613800": 23995467697.38298,
      "1641614100": 23495370573.404255,
      "1641614400": 32619585727.925533,
      "1641614700": 50497179718.44681,
      "1641615000": 67702577750.96809,
      "1641615300": 60069863926.489365,
      "1641615600": 35323577997.510635,
      "1641615900": 24773699671.531914,
      "1641616200": 26033296162.053192,
      "1641616500": 22168901492.074467,
      "1641616800": 23582663230.595745,
      "1641617100": 25922546563.61702,
      "1641617400": 24690713806.638298,
      "1641617700": 22137753611.659573,
      "1641618000": 30763230732.68085,
      "1641618300": 35937725860.702126,
      "1641618600": 53287098815.723404,
      "1641618900": 37412624353.74468,
      "1641619200": 27185680586.265957,
      "1641619500": 24002234854.287235,
      "1641619800": 23097300347.30851,
      "1641620100": 23327976088.82979,
      "1641620400": 22976781749.851063,
      "1641620700": 25886406672.37234,
      "1641621000": 24737403454.393616,
      "1641621300": 23295190483.914894,
      "1641621600": 28923377184.93617,
      "1641621900": 31075515429.457447,
      "1641622200": 31702955471.47872,
      "1641622500": 30019567667,
      "1641622800": 23845496361.02128,
      "1641623100": 23091751081.042553,
      "1641623400": 22755586986.56383,
      "1641623700": 22894006747.085106,
      "1641624000": 21873523342.606384,
      "1641624300": 23323524331.62766,
      "1641624600": 21695814130.648937,
      "1641624900": 21905042241.67021,
      "1641625200": 24325235809.69149,
      "1641625500": 27683570546.712765,
      "1641625800": 33631204978.734043,
      "1641626100": 32826612660.755318,
      "1641626400": 24016981770.276596,
      "1641626700": 22965990981.29787,
      "1641627000": 22639599705.81915,
      "1641627300": 22585638163.340427,
      "1641627600": 22076417190.861702,
      "1641627900": 24453256405.88298,
      "1641628200": 22871659861.904255,
      "1641628500": 22705675660.925533,
      "1641628800": 25854917094.446808,
      "1641629100": 28213038074.968086,
      "1641629400": 44751451324.489365,
      "1641629700": 47027175376.510635,
      "1641630000": 44649656203.53191,
      "1641630300": 21754609018.053192,
      "1641630600": 34582677660.07446,
      "1641630900": 22474280375.595745,
      "1641631200": 23152775416.11702,
      "1641631500": 22623036544.138298,
      "1641631800": 23027016496.659573,
      "1641632100": 21968854878.18085,
      "1641632400": 25726479634.20213,
      "1641632700": 27130097436.223404,
      "1641633000": 39349769619.24468,
      "1641633300": 35508686001.76596,
      "1641633600": 34765475249.78723,
      "1641633900": 22613836348.80851,
      "1641634200": 23632334334.32979,
      "1641634500": 19760820361.851063,
      "1641634800": 21393178219.87234,
      "1641635100": 25478351156.393616,
      "1641635400": 25766049439.914894,
      "1641635700": 22637000033.43617,
      "1641636000": 28112874278.957447,
      "1641636300": 25919759092.97872,
      "1641636600": 27880771415,
      "1641636900": 26298275368.52128,
      "1641637200": 23354861324.042553,
      "1641637500": 22998740072.56383,
      "1641637800": 26006165376.085106,
      "1641638100": 24153216356.106384,
      "1641638400": 22600260346.62766,
      "1641638700": 24723975076.148937,
      "1641639000": 25419601112.67021,
      "1641639300": 24789987583.69149,
      "1641639600": 45649453555.21277,
      "1641639900": 53154991242.73404,
      "1641640200": 65789350985.75532,
      "1641640500": 35493017106.776596,
      "1641640800": 26872629677.79787,
      "1641641100": 25741623817.31915,
      "1641641400": 23244857210.340427,
      "1641641700": 25602940529.861702,
      "1641642000": 26857749335.38298,
      "1641642300": 28666969412.904255,
      "1641642600": 25737337740.425533,
      "1641642900": 25037369565.446808,
      "1641643200": 32608296747.468086,
      "1641643500": 33342415579.98936,
      "1641643800": 50410446469.510635,
      "1641644100": 31706520956.531914,
      "1641644400": 28697253125.553192,
      "1641644700": 25257216657.074467,
      "1641645000": 39394805898.59574,
      "1641645300": 42071667470.11702,
      "1641645600": 86558021695.13829,
      "1641645900": 125268517079.15958,
      "1641646200": 139254583168.68085,
      "1641646500": 132968954115.20213,
      "1641646800": 122475212075.2234,
      "1641647100": 110492526646.74467,
      "1641647400": 108641480163.26596,
      "1641647700": 108952583589.28723,
      "1641648000": 108269679267.30852,
      "1641648300": 107752509570.32979,
      "1641648600": 109113386123.35106,
      "1641648900": 107762114080.37234,
      "1641649200": 107879278523.89362,
      "1641649500": 108639912844.91489,
      "1641649800": 108237224834.93617,
      "1641650100": 108718529233.45744,
      "1641650400": 115585340741.47873,
      "1641650700": 109377723209.5,
      "1641651000": 110235666414,
      "1641651300": 109206689611,
      "1641651600": 99062510092,
      "1641651900": 87105825175.5,
      "1641652200": 86497619616.5,
      "1641652500": 83945778213,
      "1641652800": 83604021733.5,
      "1641653100": 83968161657,
      "1641653400": 83375491171.5,
      "1641653700": 79853237675.5,
      "1641654000": 69140429392.5,
      "1641654300": 51157300653.5,
      "1641654600": 43594825638.5,
      "1641654900": 37001912808,
      "1641655200": 37457818391,
      "1641655500": 40028789826,
      "1641655800": 38319851933.5,
      "1641656100": 37808259275.5,
      "1641656400": 35198827051.5,
      "1641656700": 36667536834,
      "1641657000": 38033587464,
      "1641657300": 38808415348,
      "1641657600": 40587664673,
      "1641657900": 39641660187,
      "1641658200": 40648544886.5,
      "1641658500": 40296236529.5,
      "1641658800": 38817324667.5,
      "1641659100": 38624459152,
      "1641659400": 44145529709,
      "1641659700": 40441835847.5,
      "1641660000": 39651097518.5,
      "1641660300": 40734699025.5,
      "1641660600": 40600348237,
      "1641660900": 40381989370,
      "1641661200": 37520050353.5,
      "1641661500": 42542972561,
      "1641661800": 40552434484.5,
      "1641662100": 46853583739.5,
      "1641662400": 40655029863.5,
      "1641662700": 37284760988.5,
      "1641663000": 42016207387.5,
      "1641663300": 40107052640,
      "1641663600": 45740728619,
      "1641663900": 42520029942,
      "1641664200": 41141157106.5,
      "1641664500": 39605408530.5,
      "1641664800": 44270390320,
      "1641665100": 43400412765,
      "1641665400": 45285814914,
      "1641665700": 43947040506,
      "1641666000": 43436614681.5,
      "1641666300": 41258122852,
      "1641666600": 40720618855.5,
      "1641666900": 42601294942.5,
      "1641667200": 41489224887.5,
      "1641667500": 43398194256,
      "1641667800": 42268308980.5,
      "1641668100": 39181908295.5,
      "1641668400": 43996881333.5,
      "1641668700": 44260140700,
      "1641669000": 43039792958,
      "1641669300": 43893423807,
      "1641669600": 39057763906,
      "1641669900": 42425219310.5,
      "1641670200": 62800590281,
      "1641670500": 85996492017,
      "1641670800": 103360158077,
      "1641671100": 109066772759,
      "1641671400": 132323845944.5,
      "1641671700": 134686180435.5,
      "1641672000": 129297397269,
      "1641672300": 121077993155.5,
      "1641672600": 118405909902.5,
      "1641672900": 117603677197.5,
      "1641673200": 116682605441,
      "1641673500": 115521702474,
      "1641673800": 116370087763,
      "1641674100": 115802254322.5,
      "1641674400": 115360328665.5,
      "1641674700": 116392216154,
      "1641675000": 115926567424.5,
      "1641675300": 115761764791.5,
      "1641675600": 114846910090,
      "1641675900": 115556845978,
      "1641676200": 115038658364.5,
      "1641676500": 115139365869,
      "1641676800": 107810925928,
      "1641677100": 96836148342,
      "1641677400": 93533810312.5,
      "1641677700": 91303446798.5,
      "1641678000": 90339992858,
      "1641678300": 90851281012,
      "1641678600": 90438479205.5,
      "1641678900": 87855368502.5,
      "1641679200": 77490234994.5,
      "1641679500": 56407101134.5,
      "1641679800": 66395993606,
      "1641680100": 47569796344,
      "1641680400": 45365960088,
      "1641680700": 39644722068.5,
      "1641681000": 37328892793,
      "1641681300": 35736731235.5,
      "1641681600": 34194672084,
      "1641681900": 35226930637,
      "1641682200": 36580441771,
      "1641682500": 35863582968.5,
      "1641682800": 39047345721.5,
      "1641683100": 38701919482.5,
      "1641683400": 52787757755.5,
      "1641683700": 38881477983.5,
      "1641684000": 35344612667,
      "1641684300": 32601279104,
      "1641684600": 34153671168.5,
      "1641684900": 35635573617.5,
      "1641685200": 33999262666.5,
      "1641685500": 34506057176,
      "1641685800": 30912857785.5,
      "1641686100": 33314572578
    }

Sum of above raw data points is not matching with any of the above calls’ 1641600000 timestamp value.

megha10 commented 2 years ago

version details - curl http://opentsdb-read.run:4242/api/version | jq '.' { "short_revision": "b89fded4", "repo": "/opentsdb/build", "host": "558eb9594a36", "version": "2.4.1-SNAPSHOT", "full_revision": "b89fded4ee326dc064b9d7e471e9f29f7d1dede9", "repo_status": "MODIFIED", "branch": "master", "timestamp": "1625123717" }