search

vmware-archive / kafka-trigger

Kubernetes CRD controller for Kafka topic as event source for Kubeless functions
Apache License 2.0
28 stars 34 forks source link

Update sarama and cache function port #32

Closed sepetrov closed 3 years ago

sepetrov commented 3 years ago

Issue Ref: https://github.com/kubeless/kubeless/issues/826

Description:

It has been reported that the performance of the kafka-trigger-controller is limited to 5qps.

To speed up the processing, it has been suggested to cache the request to k8s API server fetching the Kubeless function port.

Added

Removed

Changed

TODOs:

sepetrov commented 3 years ago

With the above changes I managed to increase the throughput almost 4 times.

Test Environment:

Test Scenario

k6 -> nginx ingress -> http handler -> kafka -> kafka-trigger-controller -> kafka handler

Test Case: image kafka-trigger-controller v1.0.4

Load test started on 2020-09-25T14:20:25Z. The last message was published from the http handleron 2020-09-25T14:23:25Z, and the last message was consumed by the kafka handler on 2020-09-25T14:33:19Z. This makes for 10 mins delay in processing of kafka messages.

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     script: /Users/stan/Projects/typeform/kubeless-poc/function/insights/load-test.js
     output: -

  scenarios: (100.00%) 1 executors, 10 max VUs, 3m30s max duration (incl. graceful stop):
           * default: 10 looping VUs for 3m0s (gracefulStop: 30s)

running (3m00.5s), 00/10 VUs, 3884 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs  3m0s

    ✓ response status is 200

    checks.....................: 100.00% ✓ 3884 ✗ 0
    data_received..............: 1.1 MB  5.9 kB/s
    data_sent..................: 1.0 MB  5.7 kB/s
    http_req_blocked...........: avg=15.92µs  min=2µs     med=5µs      max=3.47ms p(90)=7µs      p(95)=10µs
    http_req_connecting........: avg=6.07µs   min=0s      med=0s       max=1.76ms p(90)=0s       p(95)=0s
    http_req_duration..........: avg=463.9ms  min=73.63ms med=363.57ms max=3.1s   p(90)=856.56ms p(95)=1.25s
    http_req_receiving.........: avg=97.52µs  min=28µs    med=84µs     max=5.55ms p(90)=137.7µs  p(95)=164µs
    http_req_sending...........: avg=34.35µs  min=10µs    med=30µs     max=971µs  p(90)=45µs     p(95)=60µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=463.77ms min=73.43ms med=363.43ms max=3.1s   p(90)=856.4ms  p(95)=1.25s
    http_reqs..................: 3884    21.522489/s
    iteration_duration.........: avg=464.17ms min=74.02ms med=363.82ms max=3.1s   p(90)=856.82ms p(95)=1.25s
    iterations.................: 3884    21.522489/s
    vus........................: 10      min=10 max=10
    vus_max....................: 10      min=10 max=10

Test Case: image kafka-trigger-controller #32

The requests sent by k6 were going virtually immediatelly from the http handler though the kafka-trigger-controller to the kafka handler.

          /\      |‾‾|  /‾‾/  /‾/
     /\  /  \     |  |_/  /  / /
    /  \/    \    |      |  /  ‾‾\
   /          \   |  |‾\  \ | (_) |
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     script: load-test.js
     output: -

  scenarios: (100.00%) 1 executors, 10 max VUs, 3m30s max duration (incl. graceful stop):
           * default: 10 looping VUs for 3m0s (gracefulStop: 30s)

running (3m00.5s), 00/10 VUs, 3377 complete and 0 interrupted iterations
default ✓ [======================================] 10 VUs  3m0s

    ✓ response status is 200

    checks.....................: 100.00% ✓ 3377 ✗ 0
    data_received..............: 918 kB  5.1 kB/s
    data_sent..................: 892 kB  4.9 kB/s
    http_req_blocked...........: avg=18.88µs  min=2µs     med=5µs      max=4.52ms p(90)=9µs      p(95)=15µs
    http_req_connecting........: avg=10.09µs  min=0s      med=0s       max=3.81ms p(90)=0s       p(95)=0s
    http_req_duration..........: avg=533.6ms  min=93.21ms med=477.09ms max=1.99s  p(90)=879.26ms p(95)=1.04s
    http_req_receiving.........: avg=107.34µs min=28µs    med=88µs     max=1.79ms p(90)=164µs    p(95)=202.19µs
    http_req_sending...........: avg=38.94µs  min=13µs    med=31µs     max=2.15ms p(90)=56µs     p(95)=77µs
    http_req_tls_handshaking...: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s
    http_req_waiting...........: avg=533.46ms min=93.11ms med=476.97ms max=1.99s  p(90)=879.13ms p(95)=1.04s
    http_reqs..................: 3377    18.708299/s
    iteration_duration.........: avg=533.9ms  min=93.42ms med=477.36ms max=1.99s  p(90)=879.5ms  p(95)=1.04s
    iterations.................: 3377    18.708299/s
    vus........................: 10      min=10 max=10
    vus_max....................: 10      min=10 max=10
andresmgot commented 3 years ago

It makes sense to release a new version for this right @sepetrov ? (v1.0.5)

sepetrov commented 3 years ago

@andresmgot thanks! Yes, I would suggest new release :)

andresmgot commented 3 years ago

published: https://github.com/kubeless/kafka-trigger/releases/tag/v1.0.5