get more precise latency for scale up(cold startup)

[chzhyang]

Changes

• Add iteration scale up(cold startup) for each knative service, get a latency list
• Get average, min, max and percentiles latency by parsing latency list, and write them to output(CSV, JSON and HTML)
• Print latency of each iteration when verbose is ture
• Optimize to speed up scaling down to zero in each iteration
  • updateAllowZeroInitialScale() configs allow-zero-initial-scale in configmaps/config-autoscaler before scale up, and restore it after scale up
  • updateKsvc() configs stable window and initial scale
• Add 3 flags
  • iterations: the number of iterations to scale up the service
  • time-interval: the time interval of each scale up iteration, recommend to set it no less than the sum of the stable window and cold startup time
  • stable-window: stable window per revision
  • update the docs for scale

Docs

1. generate two knative service for test: kperf service generate -n 2 -b 1 -c 1 -i 10 --namespace default --svc-prefix ktest

2. time-interval > stable window(6s is the minimal value) + cold startup latency of workload(2s~4s, helloworld-go in kperf) , so I recommand to set time-interval=10s

3. test: kperf service scale --namespace default --svc-prefix ktest --range 0,2 --verbose --output /tmp -i 20 -T 10s -s 6s

   kperf service scale --namespace default --svc-prefix ktest --range 0,2  --verbose --output /tmp -i 20 -T 10s -s 6s
   scale up service default/ktest-1 in 20 iterations:
   scale up service default/ktest-0 in 20 iterations:
   ======================= service default/ktest-1 result =====================
   iteration    0, service latency: 2.846866 s, deployment latency: 0.027383 s
   iteration    1, service latency: 2.022447 s, deployment latency: 0.023819 s
   iteration    2, service latency: 2.048867 s, deployment latency: 0.021972 s
   iteration    3, service latency: 2.057233 s, deployment latency: 0.024831 s
   iteration    4, service latency: 2.060391 s, deployment latency: 0.021583 s
   iteration    5, service latency: 2.083872 s, deployment latency: 0.024700 s
   iteration    6, service latency: 2.062645 s, deployment latency: 0.023519 s
   iteration    7, service latency: 2.050262 s, deployment latency: 0.022895 s
   iteration    8, service latency: 2.070272 s, deployment latency: 0.023013 s
   iteration    9, service latency: 2.077341 s, deployment latency: 0.024624 s
   iteration   10, service latency: 2.052486 s, deployment latency: 0.026559 s
   iteration   11, service latency: 2.061285 s, deployment latency: 0.023327 s
   iteration   12, service latency: 2.071284 s, deployment latency: 0.022161 s
   iteration   13, service latency: 2.078397 s, deployment latency: 0.025589 s
   iteration   14, service latency: 2.056636 s, deployment latency: 0.023335 s
   iteration   15, service latency: 2.087496 s, deployment latency: 0.023839 s
   iteration   16, service latency: 2.067181 s, deployment latency: 0.024321 s
   iteration   17, service latency: 2.058976 s, deployment latency: 0.023324 s
   iteration   18, service latency: 2.057849 s, deployment latency: 0.022988 s
   iteration   19, service latency: 2.081246 s, deployment latency: 0.023720 s
   service latency result:
   average: 2.102652 s
   min:     2.022447 s
   max:     2.846866 s
   p50:     2.061285 s
   p90:     2.083872 s
   p95:     2.087496 s
   p99:     2.467181 s
   deployment latency result:
   average: 0.023875 s
   min:     0.021583 s
   max:     0.027383 s
   p50:     0.023519 s
   p90:     0.025589 s
   p95:     0.026559 s
   p99:     0.026971 s
   ======================= service default/ktest-0 result =====================
   iteration    0, service latency: 2.885382 s, deployment latency: 0.072050 s
   iteration    1, service latency: 2.989419 s, deployment latency: 0.047398 s
   iteration    2, service latency: 2.050349 s, deployment latency: 0.021817 s
   iteration    3, service latency: 2.051122 s, deployment latency: 0.024518 s
   iteration    4, service latency: 2.070070 s, deployment latency: 0.021189 s
   iteration    5, service latency: 2.075538 s, deployment latency: 0.023571 s
   iteration    6, service latency: 2.074330 s, deployment latency: 0.024800 s
   iteration    7, service latency: 2.067129 s, deployment latency: 0.030077 s
   iteration    8, service latency: 2.090450 s, deployment latency: 0.022437 s
   iteration    9, service latency: 2.068713 s, deployment latency: 0.024025 s
   iteration   10, service latency: 2.062881 s, deployment latency: 0.023880 s
   iteration   11, service latency: 2.070932 s, deployment latency: 0.021486 s
   iteration   12, service latency: 2.081731 s, deployment latency: 0.024668 s
   iteration   13, service latency: 2.072034 s, deployment latency: 0.027175 s
   iteration   14, service latency: 2.066353 s, deployment latency: 0.021117 s
   iteration   15, service latency: 2.087384 s, deployment latency: 0.024184 s
   iteration   16, service latency: 2.072947 s, deployment latency: 0.027255 s
   iteration   17, service latency: 2.073980 s, deployment latency: 0.028488 s
   iteration   18, service latency: 2.063301 s, deployment latency: 0.027137 s
   iteration   19, service latency: 2.075841 s, deployment latency: 0.024166 s
   service latency result:
   average: 2.157494 s
   min:     2.050349 s
   max:     2.989419 s
   p50:     2.072034 s
   p90:     2.090450 s
   p95:     2.885382 s
   p99:     2.937401 s
   deployment latency result:
   average: 0.028072 s
   min:     0.021117 s
   max:     0.072050 s
   p50:     0.024184 s
   p90:     0.030077 s
   p95:     0.047398 s
   p99:     0.059724 s
   failed to get Knative Eventing version: namespaces "knative-eventing" not found
   Measurement saved in CSV file /tmp/20230203093607_ksvc_scaling_time.csv
   Visualized measurement saved in HTML file /tmp/20230203093607_ksvc_scaling_time.html
   Measurement saved in JSON file /tmp/20230203093607_ksvc_scaling_time.json

[codecov[bot]]

Codecov Report

Merging #338 (264d30f) into main (2d661cd) will decrease coverage by 1.08%. The diff coverage is 44.00%.

@@            Coverage Diff             @@
##             main     #338      +/-   ##
==========================================
- Coverage   50.34%   49.27%   -1.08%     
==========================================
  Files          13       13              
  Lines        2014     2145     +131     
==========================================
+ Hits         1014     1057      +43     
- Misses        935     1018      +83     
- Partials       65       70       +5     

Impacted Files	Coverage Δ
pkg/command/service/scale.go	30.04% <41.93%> (-3.85%)	:arrow_down:
pkg/command/service/common.go	50.11% <45.45%> (-2.06%)	:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[knative-prow[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chzhyang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/knative-sandbox/kperf/blob/main/OWNERS)~~ [chzhyang] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[chzhyang]

Thank you for the PR. It's useful to loop the cold startup latency test for some times to get more precise result. Will you update the correspnding documents for the updates?

Thanks, addressed, docs will be updated soon.