Issue on data/graphics with Grafana -> Trickster -> Thanos query

[florianmagnin]

Hello,

I have some graphics issues since i've added trickster to my metrology stack.

Stack:

Grafana --> trickster --> thanos query --> thanos store --> s3
                                       --> thanos sidecar --> prometheus

Here the issue: Fixed time to avoid relative time and to be sure to no ask sidecar and to do always the same request:

1st refresh: 2nd refresh: 3rd refresh, looks fine: 4rd refresh, breaked again:

Default trickster conf with basics custom:

# cat /etc/trickster/trickster.conf |egrep -v '(^#|[[:blank:]]+#)'|egrep -v '^$'
[frontend]
listen_port = 9681
    cache_type = 'memory'
        max_size_bytes = 1610612736
        max_size_backoff_bytes = 16777216
[origins]
    [origins.default]
    origin_type = 'prometheus'
    origin_url = 'http://thanos-querier:9681'
    is_default = true
    cache_name = 'default'
    timeseries_ttl_secs = 86400
    timeseries_eviction_method = 'lru'

    health_check_upstream_url = '/metrics'
log_level = 'info'

(We also tried so tweaks like disabling fast forward, set 0 ttl to negative caching on 4xx or 5xx, but nothing has helped)

# trickster -version
Trickster version: 1.1.3, buildInfo: 2020-10-20T09:20:07+0000 5ebdfcdb9aa5f11db3aa9b11db1a8eb58fb2ed44, goVersion: go1.15.3, copyright: © 2018 Comcast Corporation

Other tests: 1st refresh: 2nf refresh:

The query inspector on grafana from the last refresh: inspector.txt

We have some values with the same timestamp after 03:00 but no values between 21:00 to 03:00, that explains the missing datas displayed and the buggy metric after 03:00

Duplicate values:

$ cat inspector.txt |cut -d'[' -f2 |cut -d',' -f1 |sort -n |uniq -c |sort -nr -k1
      2 1606295520
      2 1606295460
      2 1606295400
      2 1606295340
      2 1606295280
      2 1606295220
      2 1606295160
      2 1606295100
      2 1606295040
      2 1606294980
      2 1606294920
      2 1606294860
      2 1606294800
      2 1606294740
      2 1606294680
      2 1606294620
      2 1606294560
      2 1606294500
      2 1606294440
      2 1606294380
      2 1606294320
      2 1606294260
      2 1606294200
      2 1606294140
      2 1606294080
      2 1606294020
...

Missing datas are random, and sometimes the graphic is fine.

I don't see anything weird in debug log of trickster.

Do you have to idea how to fix it ?

Thanks, Regards, Florian

[solidspark]

We continue to experience this issue and we would like some attention on it.

[jranson]

All, thanks for reporting this. We do not currently have an integration environment for Thanos, but will be standing one up over the next couple of weeks, so we can take a look at this and get it root caused. Appreciate your patience!

[sharathfeb12]

We are seeing same issue too. Any updates/recommendations?

[docker98k]

We are seeing same issue too. Any updates/recommendations?