JSON API multi-user performance testing

[cocreature]

Currently our JSON API performance tests focus on the performance of individual queries.

We should also test how performance is impacted by having ~300 concurrent users and which parameters in akka-http or in our own code, e.g., database connection pool size can help.

[akshayshirahatti-da]

As part of the Multi user performance tests the following changes was carried out

• Addition of a Connection pool, results captured in https://docs.google.com/spreadsheets/d/1HNqzOlMJX0X963DSGDZAYmk3JbWJHwvCbC4TtkUP9iQ/edit?usp=sharing
• Introduction of a key hash field, the introduction of which lead to notable performance benefit. baseline

> Number of requests                                  1000 (OK=1000   KO=-     )
> Min. response time                                   487 (OK=487    KO=-     )
> Max. response time                                  4631 (OK=4631   KO=-     )
> Mean response time                                  2462 (OK=2462   KO=-     )
> Std. deviation                                       787 (OK=787    KO=-     )
> response time 90th percentile                       2507 (OK=2507   KO=-     )
> response time 95th percentile                       3002 (OK=3002   KO=-     )
> response time 99th percentile                       3637 (OK=3637   KO=-     )
> response time 99.9th percentile                     4126 (OK=4126   KO=-     )
> Mean requests/second                               3.906 (OK=3.906  KO=-     )
---- Response time distribution ------------------------------------------------
> t < 800 ms                                             7 (  1%)
> 800 ms < t < 1200 ms                                  67 (  7%)
> t > 1200 ms                                          926 ( 93%)
> failed                                                 0 (  -%)

query using hashed key field

> Number of requests                                  1000 (OK=1000   KO=-     )
> Min. response time                                   206 (OK=206    KO=-     )
> Max. response time                                  2893 (OK=2893   KO=-     )
> Mean response time                                  1074 (OK=1074   KO=-     )
> Std. deviation                                       348 (OK=348    KO=-     )
> response time 90th percentile                       1100 (OK=1100   KO=-     )
> response time 95th percentile                       1330 (OK=1330   KO=-     )
> response time 99th percentile                       1577 (OK=1577   KO=-     )
> response time 99.9th percentile                     1832 (OK=1832   KO=-     )
> Mean requests/second                                8.85 (OK=8.85   KO=-     )
---- Response time distribution ------------------------------------------------
> t < 800 ms                                           217 ( 22%)
> 800 ms < t < 1200 ms                                 422 ( 42%)
> t > 1200 ms                                          361 ( 36%)
> failed                                                 0 (  -%)

• oracle perf ci job , to report results from the Multi user gatling performance test in slack The above test runs a scenario where 100 users query a 100K ACS to run a total of 10k queries in parallel. This was the one most feasible given the docker vm instance. This publishes the below stats(avg, stddev, p90, p99, requests_per_second) to our ci slack channel

  json_oracle/fetchByKey: 131.0 400.0 7.0 733.0 232.5
  json_oracle/fetchByQuery: 615.0 489.0 611.0 1066.0 156.2

Things not covered as part of this issue.

• The multi-user performance test runs only for a single template and fixed 3 parties. More work can be done to extend this to have multiple templates and parties.
• The payload schema is also fixed and contains minimal fields. There is scope to have different sized payloads and test performance according to that.