Events publishing latency and throughput tests

devguyio commented 5 years ago

Description

We need test cases to test the "external events publishing throughput and latency". We need to have the test cases executed and analyzed ones so that we know the possible maximum assuming the performance test cluster setup.

Requirements

Provide the following tests

Name	Test	What to measure? (SLIs)	Related components
(https://github.com/kyma-project/kyma/issues/3524) Events publishing latency	Increase no of concurrent published events sent from outside the cluster	Measure sending events latency from outside the cluster for 3 percentiles (75, 90, 99), not checking actual delivery	istio,application-event-service, event-publish
(https://github.com/kyma-project/kyma/issues/3524) Events publishing throughput	Increase no of concurrent external requests sent from outside the cluster	Measure sending events throughput, not checking actual delivery	istio,application-event-service, event-publish

Run the tests against a cluster setup as provided by https://github.com/kyma-project/kyma/issues/3521
Use the test runner setup and framework as provided by https://github.com/kyma-project/kyma/issues/3522
Locate the test case sources as described here: https://github.com/kyma-project/kyma/issues/3522

Reasons

To find out Kyma SLOs, to have performance test suite available

a-thaler commented 5 years ago

Covers https://github.com/kyma-project/kyma/issues/3518

lszymik commented 5 years ago

I have created a very simple test for sending events to Kyma: K6 Examples

The setup contains:

Event catalog:

{
"provider": "Test",
"name": "Test API with Events",
"description": "Test API with Events",
"shortDescription": "Test API with Events",
"api": {
"targetUrl": "http://52.236.146.55/",
"spec": {}
},
"events": {
"spec": {
 "asyncapi": "1.0.0",
 "info": {
   "title": "Ticket Events",
   "version": "1.0.0",
   "description": "Ticket Events"
 },
 "baseTopic": "com.sap.sc",
 "topics": {
   "petCreated.v1": {
     "subscribe": {
       "summary": "New Ticket created event",
       "payload": {
         "type": "object",
         "properties": {
           "pet": {
             "type": "object",
             "required": [
               "id"
             ],
             "example": {
               "id": "4caad296-e0c5-491e-98ac-0ed118f9474e"
             },
             "properties": {
               "id": {
                 "title": "Id",
                 "description": "Ticket identifier",
                 "type": "string"
               }
             }
           }
         }
       }
     }
   }
 }
}
}
}

-Lambda code:


const request = require('request');

module.exports = { main: function (event, context) {

return new Promise((resolve, reject) => {
    const url = `${process.env.GATEWAY_URL}/get`;
    const options = {
        url: url,
    };

    sendReq(url, resolve, reject)
})

} }

function sendReq(url, resolve, reject) { request.get(url, { json: true }, (error, response, body) => { console.log(body) if(error){ resolve(error) } resolve(body) }) }

with the package:

{ "name": "calculate-promotion", "version": "0.0.1", "dependencies": { "request": "^2.85.0" } }


The first results looks like:

k6 run ./secured-send-event.js (gke_sap-cxc4coredevkymamp_europe-west3_faros-stage)

      /\      |‾‾|  /‾‾/  /‾/
 /\  /  \     |  |_/  /  / /
/  \/    \    |      |  /  ‾‾\

/ \ | |‾\ \ | () | / __ \ |_| \\ ___/ .io

execution: local output: - script: ./secured-send-event.js

duration: 1m0s, iterations: -
     vus: 25,   max: 25

done [==========================================================] 1m0s / 1m0s

✗ transaction time OK
 ↳  95% — ✓ 1307 / ✗ 64
✓ status was 200

checks.....................: 97.66% ✓ 2678 ✗ 64
data_received..............: 605 kB 10 kB/s
data_sent..................: 386 kB 6.4 kB/s
http_req_blocked...........: avg=5.85ms   min=0s      med=1µs     max=324.14ms p(90)=1µs      p(95)=1µs
http_req_connecting........: avg=654.34µs min=0s      med=0s      max=37.1ms   p(90)=0s       p(95)=0s
http_req_duration..........: avg=92.94ms  min=40.07ms med=67.83ms max=1.32s    p(90)=159.5ms  p(95)=197.74ms
http_req_receiving.........: avg=41.34µs  min=13µs    med=39µs    max=245µs    p(90)=57µs     p(95)=64.5µs
http_req_sending...........: avg=148.65µs min=53µs    med=141µs   max=1ms      p(90)=211µs    p(95)=237µs
http_req_tls_handshaking...: avg=4.62ms   min=0s      med=0s      max=257.72ms p(90)=0s       p(95)=0s
http_req_waiting...........: avg=92.75ms  min=39.92ms med=67.72ms max=1.32s    p(90)=159.34ms p(95)=197.38ms
http_reqs..................: 1371   22.8499/s
iteration_duration.........: avg=1.1s     min=1.04s   med=1.06s   max=2.33s    p(90)=1.16s    p(95)=1.2s
iterations.................: 1346   22.433235/s
vus........................: 25     min=25 max=25
vus_max....................: 25     min=25 max=25

or

k6 run ./secured-send-event.js (gke_sap-cxc4coredevkymamp_europe-west3_faros-stage)

      /\      |‾‾|  /‾‾/  /‾/
 /\  /  \     |  |_/  /  / /
/  \/    \    |      |  /  ‾‾\

/ \ | |‾\ \ | () | / __ \ |_| \\ ___/ .io

execution: local output: - script: ./secured-send-event.js

duration: 1m0s, iterations: -
     vus: 25,   max: 25

done [==========================================================] 1m0s / 1m0s

✗ status was 200
 ↳  99% — ✓ 1338 / ✗ 2
✗ transaction time OK
 ↳  88% — ✓ 1191 / ✗ 149

checks.....................: 94.36% ✓ 2529 ✗ 151
data_received..............: 593 kB 9.9 kB/s
data_sent..................: 378 kB 6.3 kB/s
http_req_blocked...........: avg=6.67ms   min=0s      med=0s      max=361.11ms p(90)=1µs      p(95)=1µs
http_req_connecting........: avg=682.78µs min=0s      med=0s      max=38.16ms  p(90)=0s       p(95)=0s
http_req_duration..........: avg=117.31ms min=42.32ms med=99.55ms max=479.33ms p(90)=206.2ms  p(95)=266.5ms
http_req_receiving.........: avg=48.54µs  min=16µs    med=39µs    max=2.69ms   p(90)=59µs     p(95)=70.04µs
http_req_sending...........: avg=154.66µs min=49µs    med=142µs   max=6.01ms   p(90)=208µs    p(95)=232µs
http_req_tls_handshaking...: avg=4.77ms   min=0s      med=0s      max=261.54ms p(90)=0s       p(95)=0s
http_req_waiting...........: avg=117.11ms min=42.18ms med=99.37ms max=479.16ms p(90)=206.02ms p(95)=266.28ms
http_reqs..................: 1340   22.333214/s
iteration_duration.........: avg=1.12s    min=1.04s   med=1.1s    max=1.7s     p(90)=1.21s    p(95)=1.28s
iterations.................: 1315   21.91655/s
vus........................: 25     min=25 max=25
vus_max....................: 25     min=25 max=25



I have been using cluster provisioned by Faros without security checks.

devguyio commented 5 years ago

I think this one should go back to TODO cause while having the k6 test is an essential step, we still have the rest of the AC:

Run the tests against a cluster setup as provided by #3521
Use the test runner setup and framework as provided by #3522
Locate the test case sources as described here: #3522

Disper commented 5 years ago

@Abd4llA do you expect any additional work from our side on it? If yes, could you list what is missing?

lszymik commented 5 years ago

We have numbers: https://grafana.perf.kyma-project.io/d/ReuNR5Aik/kyma-performance-test-results?orgId=1&from=1559029702590&to=1559030932969&var-SHAID=82d408f8&var-Measurement=All&var-component=application-gateway&var-testname=send_event_gateway_16vu_180s_1000

kyma-project / kyma