[BUG] Timestamps don't appear trimmed when using the VictoriaMetrics backend

[keliss]

Describe the bug See https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1555. I did a quick investigation, and both querying and ingestion of metrics work fine on the VM side - the timestamp gets trimmed. However, when querying metrics via Grafana using carbonapi, the metric gets the seconds part of a timestamp set to the current number of seconds (you can see it if you keep changing the time range frequently without hitting the query cache), no matter if I use the prometheus or victoriametrics protocol.

CarbonAPI Version carbonapi-0.15.4~1-1

CarbonAPI Configuration:

listen: "0.0.0.0:8182"
cache:
  type: "mem"
  size_mb: 0
  defaultTimeoutSec: 60
cpus: 0
tz: ""
graphite:
  host: "localhost:2003"
  interval: "60s"
  prefix: "carbon.api"
  pattern: "{prefix}.{fqdn}"
pidFile: ""
logger:
  - logger: ""
    file: "/var/log/carbonapi/carbonapi-1.log"
    level: "warn"
    encoding: "console"
upstreams:
  buckets: 10
  slowLogThreshold: 7s
  timeouts:
    find: "10s"
    render: "30s"
    connect: "200ms"
  keepAliveInterval: "30s"
  maxIdleConnsPerHost: 1000
  backendsv2:
    backends:
      - groupName: "victoriametrics"
        protocol: "victoriametrics"
        lbMethod: "rr"
        maxTries: 2
        backendOptions:
          step: "1m"
        servers:
          - "http://storage:8428"

Simplified query (if applicable) Any metric and query would do.

[keliss]

Also, querying carbonapi directly, without Grafana, doesn't make any difference.

[Civil]

So far I don't understand what the problem is.

carbonapi trusts backend to return valid data and uses timestamp of a first point as start: https://github.com/go-graphite/carbonapi/blob/main/zipper/protocols/victoriametrics/fetch.go#L146-L152

If it doesn't match what you are expecting - it means that VictoriaMetrics returned it like that.

[keliss]

Maybe I don't understand what API carbonapi uses to fetch the data from VM, but the Prometheus API always returns trimmed timestamps for sure, I checked that. If carbonapi relies on the same API (for example, if the "prometheus" protocol is used), then this indicates a bug in the carbonapi logic because it overrides what the backend returns.

[Civil]

Currently carbonapi uses /api/v1/query_range for VictoriaMetircs queries (as far as I understand that query should return raw data as VM stores it)

And as I've showed above - it doesn't override timestamps from the backend reply.

[keliss]

Ok, I've tried querying VM directly using this API and passing in non-trimmed start and end parameters, and I see that the backend returns timestamps proportional to the values of the boundaries (e.g. if start ends with 5 seconds, and step is 60 seconds, then the first datapoint and all the subsequent ones will also end with 5 seconds), no trimming occurs, as you said. However, when making the same requests to that API via Grafana, the result gets trimmed to 0 seconds, and when the same query is run via Grafana to carbonapi the timestamps don't get trimmed, it just shows whatever it got from the backend. Also, I don't know what the difference is, but when the same is done against a go-carbon backend (querying carbonapi via Grafana), the results do get trimmed. This is some inconsistent behavior, and I think the problem is somewhere in carbonapi because if it was in Grafana, I wouldn't see the timestamps getting trimmed with the go-carbon backend as well since the API Grafana uses to interact with carbonapi stays the same when I switch the backends.

[Civil]

With go-carbon it is not exactly trimmed. Data model of whisper actually force timestamps to be aligned to the border of the time bucket (AFAIR on receive it just substracts timestamp % step for the specific bucket from received timestamp and that's what gets stored on disk). And same goes for carbon-clickhouse (that is there ensured on carbon-clickhouse and graphite-clickhouse layers as far as I remember)

In case why in grafana you see different results - I guess there is some parameter or header that carbonapi doesn't pass, however for me it's not clear from the VMs documentation what would that be.

[Felixoid]

And same goes for carbon-clickhouse (that is there ensured on carbon-clickhouse and graphite-clickhouse layers as far as I remember)

Just as a tiny detail, cch sends the points as is and everything is aligned to timestamps on the CH side during the merging according to retentions config. So the gch have points requested with proper alignment all the time.

[Civil]

@Felixoid well, gch aligns reply for not yet merged parts.

[Felixoid]

Kinda =)

In the end, it gets aligned points directly from CH, which matches the original whisper approach. I'd say, CH aligns them.

Sorry for offtopic though