trickstercache / trickster

Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
https://trickstercache.org
Apache License 2.0
1.99k stars 177 forks source link

Slow performance #465

Open shoopdedoop opened 4 years ago

shoopdedoop commented 4 years ago

Trickster version 1.1.2

I am running a simple query against trickster.

sum(increase(rules_count_total[1w]))

It takes a long time to complete.

Total request time | 1.381 min

Even subsequent calls take the same amount of time.

If I run directly against the backend it runs must faster Total request time | 295 ms

From the logs it is taking a long time for the delta routine to complete time=2020-07-23T19:32:01.735074602Z app=trickster caller=proxy/engines/deltaproxycache.go:465 level=debug event="delta routine completed" cacheStatus=phit reqStart=1595532420 reqEnd=1595532720 extentsFetched=1595532675-1595532720 cacheKey=m3coordinator-master-monitoring-services-m3db-cluster-pv:7201.dpc.e7a9155bce62ee0498270ac8de78ef67

The upstream requests from the logs are also fast.

time=2020-07-23T19:31:40.649145208Z app=trickster caller=proxy/engines/access_logs.go:27 level=debug event="upstream request" handlerName=query_range uri=http://m3coordinator-master-monitoring-services-m3db-cluster-pv:7201/api/v1/query_range userAgent=Go-http-client/1.1 code=200 durationMS=10 originName=m3coordinator-master-monitoring-services-m3db-cluster-pv method=POST size=63 originType=prometheus

Any thoughts?

jranson commented 4 years ago

Thanks for this report - we will take a look!