perf: 会有偶发的超过平均耗时很多的请求

wklken commented 1 year ago

直接压IP 和端口号

./plow 'http://{ip}:{port}/api/bk-iam/dev/api/v1/policy/query' -c 1 -d 60s -H 'X-Bkapi-Authorization: {"bk_app_code": "x", "bk_app_secret": "y' -b @body.json  -T "application/json" -m POST

body.json

{
    "system": "x_rbac",
    "subject":
    {
        "type": "user",
        "id": "somebody"
    },
    "action": {
        "id": "xxxx_list"
    },
    "resources": [
    ]
}

大部分情况正常, 偶发的耗时很长 (概率大概是万分之五)

result

Summary:
  Elapsed        30s
  Count         1952
    2xx         1952
  RPS         65.058
  Reads    5.658MB/s
  Writes   0.028MB/s

Statistics    Min       Mean    StdDev      Max   
  Latency   13.789ms  15.358ms  3.249ms  147.258ms
  RPS        56.02     65.05     2.22      67.92  

Latency Percentile:
  P50         P75       P90       P95       P99       P99.9     P99.99  
  14.932ms  15.441ms  16.643ms  17.842ms  20.152ms  147.258ms  147.258ms

Latency Histogram:
  14.909ms   1571  80.48%
  15.934ms    228  11.68%
  17.523ms    110   5.64%
  18.622ms     26   1.33%
  19.986ms      6   0.31%
  22.96ms       5   0.26%
  25.8ms        5   0.26%
  147.258ms     1   0.05%

wklken commented 1 year ago

大概率是 redis pipeline时 cmd 过多导致的

需要关注线上, 是否需要进一步优化

如果是, 需要完整排查并确认下有没办法降低

wklken commented 1 year ago

补充, 直接压iam nodeport, 规避可能得网关影响;

/api/v2/policy/systems/bk_ci_rbac/query/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    12.56ms    4.79ms 219.89ms   98.88%
    Req/Sec    80.59      5.30    90.00     74.31%
  24143 requests in 5.00m, 2.04GB read
Requests/sec:     80.47
Transfer/sec:      6.98MB

TencentBlueKing / bk-iam-saas

perf: 会有偶发的超过平均耗时很多的请求 #2050