trickstercache / trickster

Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
https://trickstercache.org
Apache License 2.0
1.99k stars 177 forks source link

Seeing wrong values in grafana #638

Open ajayanaj opened 1 year ago

ajayanaj commented 1 year ago

Hello,

We are seeing some wrong behaviour with Trickster when using with Grafana, it's showing all the series(sometimes summing up all the series) as below.

Screenshot 2023-02-06 at 3 24 08 PM Screenshot 2023-02-06 at 3 58 30 PM

When we are connecting grafana directly to prometheus it works as expected.

Screenshot 2023-02-06 at 3 24 51 PM

The trickster is placed in front of Victoria metrics and the configuration is below.

`[frontend] listen_port = 8580 connections_limit = 0

[caches]

  [caches.default]
  cache_type = "memory"
    [caches.default.index]
    flush_interval_secs = 5
    max_size_bytes = 536870912
    max_size_backoff_bytes = 16777216
    max_size_objects = 0
    max_size_backoff_objects = 100

[origins]

  [origins.default]
  origin_type = "prometheus"
  origin_url = "vmselect:8481/select/xxx/prometheus"
  timeout_secs = 180
  keep_alive_timeout_secs = 300
  max_idle_conns = 20
  timeseries_retention_factor = 1024
  timeseries_ttl_secs = 86400
  timeseries_eviction_method = "oldest"
  fast_forward_disable = true
  fastforward_ttl_secs = 15
  max_ttl_secs = 604800

[metrics]
listen_port = 8581

[logging]
log_level = "error"`

Trickster version: 1.1.5

Not able to figure out what went wrong here, can anyone please help us here.