search

Kong / kong-plugin-prometheus

Prometheus plugin for Kong - this plugin has been moved into https://github.com/Kong/kong, please open issues and PRs in that repo
Apache License 2.0
119 stars 57 forks source link

Prometheus severely affects performance！！ #61

Closed chensunny closed 4 years ago

chensunny commented 5 years ago

Please read the CONTRIBUTING.md guidelines to learn on which channels you can seek for help and ask general questions:

https://github.com/Kong/kong/blob/master/CONTRIBUTING.md#where-to-seek-for-help

Summary

Prometheus severely affects performance

Steps To Reproduce

  1. add a service and a route then i do benchmark 
wrk -t 100 -c 100 --latency -d 60  http://{host:port}//users1/1223/profile
Running 1m test @ http://10.8.2.7:8000/users1/1223/profile
  100 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     9.54ms    5.98ms 224.42ms   91.25%
    Req/Sec   110.69     31.33   393.00     76.78%
  Latency Distribution
     50%    8.20ms
     75%    9.67ms
     90%   13.77ms
     99%   28.31ms
  662091 requests in 1.00m, 555.01MB read
Requests/sec:  11017.09
Transfer/sec:      9.24MB
  1. I add prometheus plugin 
curl -X POST --insecure   --url https://{host:port}/services/http-svc/plugins  --data "name=c"
wrk -t 100 -c 100 --latency -d 60  http://10.8.2.7:8000/users1/1223/profile
Running 1m test @ http://10.8.2.7:8000/users1/1223/profile
  100 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.03ms   14.21ms 256.80ms   84.44%
    Req/Sec    30.94     11.50   151.00     86.23%
  Latency Distribution
     50%   30.14ms
     75%   37.09ms
     90%   47.52ms
     99%   83.43ms
  185401 requests in 1.00m, 155.55MB read
Requests/sec:   3084.91
Transfer/sec:      2.59MB

08:13:22      UID       PID    %usr %system  %guest    %CPU   CPU  Command
08:13:23     1337       135   87.00   12.00    0.00   99.00     0  nginx
08:13:23     1337       136   86.00   13.00    0.00   99.00     3  nginx

performance is bad !!! and latency is high

Additional Details & Logs

guanlan commented 5 years ago

@chensunny Thanks for reporting. To help us better understand the issues, can you provide the following details:

  1. Flame graphs with and without the plugin.
  2. A sample output of /metrics
  3. How many services and routes do you have?
  4. Do you have other plugins enabled?

We've recognized a few bottlenecks of the Prometheus plugin, but it hasn't been planned on our roadmap yet.

In the meantime, I'd suggest that you try out the statd plugin to get the metrics: https://docs.konghq.com/hub/kong-inc/statsd/ https://github.com/prometheus/statsd_exporter

chensunny commented 5 years ago

i am in docker with gke (4.14.x) ,how to install Flame graphs systemtap easy ~ without systemtap can`t show lua vm stack @ @guanlan