go-graphite / carbonapi

Implementation of graphite API (graphite-web) in golang
Other
308 stars 140 forks source link

[BUG] ConsolidateBy function always uses average when maxBatchSize=0 for carbonapi_v3_pb #790

Closed perederyaev closed 1 year ago

perederyaev commented 1 year ago

Describe the bug When I have maxBatchSize=0 and upstream protocol carbonapi_v3_pb ConsolidateBy function always uses average for aggregation.

CarbonAPI Version This issue is reproducible on current master.

CarbonAPI Configuration:

listen: "0.0.0.0:8081"
upstreams:
  backendsv2:
    backends:
    - groupName: "test"
      protocol: "carbonapi_v3_pb"
      lbMethod: "broadcast"
      maxBatchSize: 0
      servers: ["http://x.x.x.x:9090"]

expireDelaySec: 0
logger:
- logger: ""
  file: "/var/log/carbonapi.log"
  level: "debug"
  encoding: "console"
  encodingTime: "iso8601"
  encodingDuration: "seconds"

Simplified query (if applicable) curl "localhost:8081/render?format=json&target=consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,'sum')&from=18%3A08_20230725&until=18%3A17_20230725&maxDataPoints=5"

Backend response (if possible)

Consolidation is not applied as maxDataPoints is greater than metrics number in the range

$ curl "localhost:8081/render?format=json&target=consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,'sum')&from=18%3A08_20230725&until=18%3A17_20230725&maxDataPoints=50" | jq

[
  {
    "target": "consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,\"sum\")",
    "datapoints": [
      [
        58,
        1690308480
      ],
      [
        59,
        1690308540
      ],
      [
        60,
        1690308600
      ],
      [
        59,
        1690308660
      ],
      [
        60,
        1690308720
      ],
      [
        59,
        1690308780
      ],
      [
        60,
        1690308840
      ],
      [
        60,
        1690308900
      ],
      [
        59,
        1690308960
      ],
      [
        59,
        1690309020
      ]
    ],
    "tags": {
      "consolidateBy": "sum",
      "name": "carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded"
    }
  }
]

maxBatchSize=0 - avg is used ignoring provided sum function

$ curl "localhost:8081/render?format=json&target=consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,'sum')&from=18%3A08_20230725&until=18%3A17_20230725&maxDataPoints=5" | jq

[
  {
    "target": "consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,\"sum\")",
    "datapoints": [
      [
        58.5,
        1690308480
      ],
      [
        59.5,
        1690308600
      ],
      [
        59.5,
        1690308720
      ],
      [
        60,
        1690308840
      ],
      [
        59,
        1690308960
      ]
    ],
    "tags": {
      "consolidateBy": "sum",
      "name": "carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded"
    }
  }
]

maxBatchSize=1 - sum is used like it should be

$ curl "localhost:8081/render?format=json&target=consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,'sum')&from=18%3A08_20230725&until=18%3A17_20230725&maxDataPoints=5" | jq
[
  {
    "target": "consolidateBy(carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded,\"sum\")",
    "datapoints": [
      [
        117,
        1690308480
      ],
      [
        119,
        1690308600
      ],
      [
        119,
        1690308720
      ],
      [
        120,
        1690308840
      ],
      [
        118,
        1690308960
      ]
    ],
    "tags": {
      "consolidateBy": "sum",
      "name": "carbon.agents.04f2e201fc3b.upload.graphite_tagged.uploaded"
    }
  }
]

Additional context If use protocol: "carbonapi_v2_pb" it works well in spite of maxBatchSize setting

perederyaev commented 1 year ago

Looks like issue is in graphite-clickhouse not in carbonapi. In our case graphite-clickhouse has internal-aggregation enabled and uses "avg" as rollup-default-function .