Create performance metrics

[simonfernandes]

Part of https://github.com/UserOfficeProject/user-office-project-issue-tracker/issues/933

After talking to the SCD cloud team (https://github.com/UserOfficeProject/user-office-project-issue-tracker/issues/938) we decided it would be useful to load test the system and create metrics. This will help us know whether we need to make any performance improvements for future rounds, and allow us to use them in future benchmarks.

Some notes from parent issue:

• Get metrics from when the system is not under load (e.g. time taken to submit a proposal) to use as a comparison for when it is
• Possibly re-run Cosimo's performance benchmark: https://github.com/UserOfficeProject/user-office-project-issue-tracker/issues/18#issuecomment-1240685423 and https://ci.devs.facilities.rl.ac.uk/view/ProposalLoadTest/
• Consider finding and using how long people actually take to submit a proposal from creation through to submission (and use average size file upload, no. co-investigators etc.) for more realistic results

[mutambaraf]

Steps to get the the performance matrix working in https://github.com/isisbusapps/internal-scripts/blob/master/proposals-load-tester.

• [x] Update and rebuild tests by running Dev_Build_ProposalLoadTest
• [x] Update the call Performance test in dev and prod
• [x] Make changes to the load test kubeconfig

[mutambaraf]

We can also explore K6 load testing it does the same tests but with minimum configurations. Example tests are here Example and they use almost the same language as JUnit or Cypress

[mutambaraf]

@simonfernandes @srconway I have re-run Cosimo's performance benchmark: https://github.com/UserOfficeProject/user-office-project-issue-tracker/issues/18#issuecomment-1240685423 and https://ci.devs.facilities.rl.ac.uk/view/ProposalLoadTest/ Here are the results	concurrent	successes	failures*	% failure	mean	median	90th	duration	submissions/min
10	29	1	3.3	51.28	50.581	55.378	305	5.7
20	59	1	1.7	56.95	64.8	56.79	345	10.26
30	89	1	1.1	58.2	56.75	66.66	534	10
40	111	9	7.5	57.8	56.75	65.87	683	9.75
50	149	1	0.67	57.51	56.25	65.76	837	10.68
60	168	12	6.7	57.57	56.57	64.747	997	10.11

Failures

Details on the failures for each run. See also the proposals backend errors and failed requests processed by the Ingress controller during the test period.

The logs and screenshots for all these can be found in the results of the Jenkins run.

10

• 1 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > Network request failed\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"

20

• 1 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > GraphQL Error (Code: 502) {\"response\":{\"error\":\"\r\n\r\n\r\n502 Bad Gateway\r\nnginx\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\",\"status\":502,\"headers\":{\"map\":{\"cache-control\":\"max-age=0,private\",\"connection\":\"keep-alive\",\"content-encoding\":\"gzip\",\"content-type\":\"text/html\",\"date\":\"Tue, 19 Mar 2024 12:44:56 GMT\",\"keep-alive\":\"timeout=5, max=99\",\"server\":\"Apache/2.4.58 (Win64) OpenSSL/3.1.3\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains, max-age=15724800; includeSubDomains\",\"transfer-encoding\":\"chunked\",\"vary\":\"Accept-Encoding\"}}},\"request\":{\"query\":\"query getCall($callId: Int!) {\n call(callId: $callId) {\n ...call\n }\n}\n\nfragment call on Call {\n id\n shortCode\n startCall\n endCall\n endCallInternal\n startReview\n endReview\n startFapReview\n endFapReview\n startNotify\n endNotify\n startCycle\n endCycle\n cycleComment\n surveyComment\n referenceNumberFormat\n proposalWorkflowId\n templateId\n esiTemplateId\n pdfTemplateId\n allocationTimeUnit\n instruments {\n id\n name\n shortCode\n description\n availabilityTime\n submitted\n fapId\n fap {\n id\n code\n }\n scientists {\n ...basicUserDetails\n }\n }\n faps {\n id\n code\n }\n proposalWorkflow {\n id\n name\n description\n }\n template {\n templateId\n name\n isArchived\n }\n proposalCount\n title\n description\n submissionMessage\n isActive\n isActiveInternal\n}\n\nfragment basicUserDetails on BasicUserDetails {\n id\n firstname\n lastname\n preferredname\n institution\n institutionId\n position\n created\n placeholder\n email\n}\",\"variables\":{\"callId\":54}}}\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"

30

• 1 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > GraphQL Error (Code: 502) {\"response\":{\"error\":\"\r\n\r\n\r\n502 Bad Gateway\r\nnginx\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\",\"status\":502,\"headers\":{\"map\":{\"cache-control\":\"max-age=0,private\",\"connection\":\"keep-alive\",\"content-encoding\":\"gzip\",\"content-type\":\"text/html\",\"date\":\"Tue, 19 Mar 2024 12:44:56 GMT\",\"keep-alive\":\"timeout=5, max=99\",\"server\":\"Apache/2.4.58 (Win64) OpenSSL/3.1.3\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains, max-age=15724800; includeSubDomains\",\"transfer-encoding\":\"chunked\",\"vary\":\"Accept-Encoding\"}}},\"request\":{\"query\":\"query getCall($callId: Int!) {\n call(callId: $callId) {\n ...call\n }\n}\n\nfragment call on Call {\n id\n shortCode\n startCall\n endCall\n endCallInternal\n startReview\n endReview\n startFapReview\n endFapReview\n startNotify\n endNotify\n startCycle\n endCycle\n cycleComment\n surveyComment\n referenceNumberFormat\n proposalWorkflowId\n templateId\n esiTemplateId\n pdfTemplateId\n allocationTimeUnit\n instruments {\n id\n name\n shortCode\n description\n availabilityTime\n submitted\n fapId\n fap {\n id\n code\n }\n scientists {\n ...basicUserDetails\n }\n }\n faps {\n id\n code\n }\n proposalWorkflow {\n id\n name\n description\n }\n template {\n templateId\n name\n isArchived\n }\n proposalCount\n title\n description\n submissionMessage\n isActive\n isActiveInternal\n}\n\nfragment basicUserDetails on BasicUserDetails {\n id\n firstname\n lastname\n preferredname\n institution\n institutionId\n position\n created\n placeholder\n email\n}\",\"variables\":{\"callId\":54}}}\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"

40

• 5 "Timed out retrying after 60000ms: Expected to find element: [aria-label=\"New Proposal\"], but never found it."
• 3 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > Network request failed\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"
• 1 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > Cannot read properties of null (reading '1'): {\"response\":{\"errors\":[{\"message\":\"Cannot read properties of null (reading '1')\",\"locations\":[{\"line\":2,\"column\":3}],\"path\":[\"users\"],\"extensions\":{\"code\":\"INTERNAL_SERVER_ERROR\",\"stacktrace\":[\"TypeError: Cannot read properties of null (reading '1')\",\" at /home/node/app/build/src/datasources/stfc/UOWSSoapInterface.js:1815:195\",\" at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\"]}}],\"data\":{\"users\":null},\"status\":200,\"headers\":{\"map\":{\"access-control-allow-origin\":\"*\",\"cache-control\":\"max-age=0,private\",\"connection\":\"keep-alive\",\"content-encoding\":\"gzip\",\"content-type\":\"application/json; charset=utf-8\",\"date\":\"Tue, 19 Mar 2024 13:10:56 GMT\",\"etag\":\"W/\\"1af-IpeY/o5iP0Iy0yHlTvPhSkQNQw4\\"\",\"keep-alive\":\"timeout=5, max=95\",\"server\":\"Apache/2.4.58 (Win64) OpenSSL/3.1.3\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains, max-age=15724800; includeSubDomains\",\"transfer-encoding\":\"chunked\",\"vary\":\"Accept-Encoding\",\"x-powered-by\":\"Express\"}}},\"request\":{\"query\":\"query getUsers($filter: String, $first: Int, $offset: Int, $userRole: UserRole, $subtractUsers: [Int!], $orderBy: String, $orderDirection: String) {\n users(\n filter: $filter\n first: $first\n offset: $offset\n userRole: $userRole\n subtractUsers: $subtractUsers\n orderBy: $orderBy\n orderDirection: $orderDirection\n ) {\n users {\n ...basicUserDetails\n }\n totalCount\n }\n}\n\nfragment basicUserDetails on BasicUserDetails {\n id\n firstname\n lastname\n preferredname\n institution\n institutionId\n position\n created\n placeholder\n email\n}\",\"variables\":{\"filter\":\"\",\"offset\":0,\"subtractUsers\":[],\"first\":10,\"userRole\":\"USER\",\"orderBy\":\"\",\"orderDirection\":\"\"}}}\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"

50

• 1 "The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > Network request failed\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application"

60

• 10 Timed out retrying after 60000ms: Expected to find element: [aria-label=\"New Proposal\"], but never found it.
• 2 The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.\n\n > Network request failed\n\nWhen Cypress detects uncaught errors originating from your application it will automatically fail the current test.\n\nThis behavior is configurable, and you can choose to turn this off by listening to the uncaught:exception event.\n\nhttps://on.cypress.io/uncaught-exception-from-application

Analysis

The system is clearly performing above its previous metrics when looking at mean,median,90th ,duration and submissions/min in the previous tests and a good example is at 50 users which does point to some issues outside the system causing some of the timeout this can be related to infrastructure , configuration or network issues. The issue of dashboard timeout has been raise before and k6 tests are also picking this problem in the system. Users are able to login under most fail cases but the calls are not being loaded in time for the tests to proceed while we also have a reasonable wait time of 60000ms (1 min) .We definitely need to rule out any issues of communication between frontend and backend before looking at the communication between the backend and users system/database.

May l also further recommend that we create issues for solutions that where raised in the previous tests and furthermore the performance issues that we experienced in the last ISIS Direct 24-1 looks like they where not related to the system itself but other services we are consuming.

[mutambaraf]

Dev logs are currently down at the moment

[mutambaraf]

Pod Metrics during 18/04/2024

Our system unfortunately showed some performance issues during this peak period but l managed to capture some backend and frontend pod metrics which l think will help us decide which area we need to investigate further.

Backend

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     1064999396n
    Memory:  2456144Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T14:58:11Z
Timestamp:             2024-04-17T14:58:03Z
Window:                15.03s

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     1090321231n
    Memory:  1102788Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:05:03Z
Timestamp:             2024-04-17T15:05:03Z
Window:                15.034s
Events:                <none>

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     163627456n
    Memory:  1000056Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:05:52Z
Timestamp:             2024-04-17T15:05:48Z
Window:                14.962s
Events:                <none>

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     403584443n
    Memory:  879092Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:06:56Z
Timestamp:             2024-04-17T15:06:48Z
Window:                14.956s

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     1440245262n
    Memory:  2799252Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:08:39Z
Timestamp:             2024-04-17T15:08:33Z
Window:                15.029s
Events:                <none>

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     1124768291n
    Memory:  888980Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:11:48Z
Timestamp:             2024-04-17T15:11:48Z
Window:                14.966s
Events:                <none>

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     376270104n
    Memory:  904692Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:26:10Z
Timestamp:             2024-04-17T15:26:03Z
Window:                14.964s
Events:                <none>

Name:         proposal-backend-54fc9877df-nbhcx
Namespace:    apps
Labels:       app=proposal-backend
              pod-template-hash=54fc9877df
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-backend
  Usage:
    Cpu:     3390601n
    Memory:  389168Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T16:18:11Z
Timestamp:             2024-04-17T16:18:03Z
Window:                15.022s
Events:                <none>

Frontend

Name:         proposal-frontend-68c65d4749-dmncj
Namespace:    apps
Labels:       app=proposal-frontend
              pod-template-hash=68c65d4749
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  frontend
  Usage:
    Cpu:     377285n
    Memory:  10656Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T14:58:55Z
Timestamp:             2024-04-17T14:58:48Z
Window:                15.009s

Name:         proposal-frontend-68c65d4749-dmncj
Namespace:    apps
Labels:       app=proposal-frontend
              pod-template-hash=68c65d4749
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-frontend
  Usage:
    Cpu:     410694n
    Memory:  10624Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:04:12Z
Timestamp:             2024-04-17T15:04:03Z
Window:                14.949s
Events:                <none>

Name:         proposal-frontend-68c65d4749-dmncj
Namespace:    apps
Labels:       app=proposal-frontend
              pod-template-hash=68c65d4749
Annotations:  <none>
API Version:  metrics.k8s.io/v1beta1
Containers:
  Name:  proposal-frontend
  Usage:
    Cpu:     109761n
    Memory:  10692Ki
Kind:        PodMetrics
Metadata:
  Creation Timestamp:  2024-04-17T15:06:23Z
Timestamp:             2024-04-17T15:06:18Z
Window:                15.029s
Events:                <none>

Across the cluster

NAME                                       CPU(cores)   MEMORY(bytes)
approved-experiments-759cf565c4-cpn5t      0m           89Mi
auth-service-69fc9b6fc-fb8w2               1m           391Mi
clam-antivirus-5966d8867b-p749f            1m           1693Mi
cron-service-794c45ff7-fzxrr               1m           301Mi
gitea-d9bc6786d-s48r7                      10m          157Mi
mailchimp-webhooks-5bf5cb7459-wp8lt        1m           219Mi
merge-user-tool-6b49fdbb54-rl2tb           2m           315Mi
messages-service-8b9687ccb-l2sqg           3m           269Mi
panel-member-manager-775ddb9fbb-9bq4m      0m           64Mi
proposal-allocations-67f9df44c-wkgmf       4m           741Mi
proposal-backend-54fc9877df-nbhcx          191m         477Mi
proposal-factory-cf5c678bd-pbggn           94m          288Mi
proposal-frontend-68c65d4749-dmncj         16m          10Mi
proposal-launchpad-69cccb94d8-sn2zd        1m           78Mi
proposal-lookup-service-86cf77cd87-tbqsv   6m           1178Mi
rabbitmq-0                                 5m           143Mi
request-consumables-697bd8cd5b-w7m7d       0m           82Mi
schedule-service-57dc7b5f89-jdk2w          10m          1964Mi
settings-tool-service-58fc879c7c-5tzmm     5m           590Mi
trino-d76869bb-wwhvt                       11m          3435Mi
user-office-web-service-55f8bf87c7-qlj7b   19m          1300Mi
user-wifi-login-service-6564fb8796-vpzn5   3m           726Mi
visits-service-75f95fc5f6-7ftql            11m          1373Mi

NAME                                       CPU(cores)   MEMORY(bytes)
approved-experiments-759cf565c4-cpn5t      0m           89Mi
auth-service-69fc9b6fc-fb8w2               1m           391Mi
clam-antivirus-5966d8867b-p749f            9m           1693Mi
cron-service-794c45ff7-fzxrr               1m           301Mi
gitea-d9bc6786d-s48r7                      2m           157Mi
mailchimp-webhooks-5bf5cb7459-wp8lt        1m           219Mi
merge-user-tool-6b49fdbb54-rl2tb           1m           315Mi
messages-service-8b9687ccb-l2sqg           2m           269Mi
panel-member-manager-775ddb9fbb-9bq4m      0m           64Mi
proposal-allocations-67f9df44c-wkgmf       1m           742Mi
proposal-backend-54fc9877df-nbhcx          277m         528Mi
proposal-factory-cf5c678bd-pbggn           103m         287Mi
proposal-frontend-68c65d4749-dmncj         0m           10Mi
proposal-launchpad-69cccb94d8-sn2zd        1m           78Mi
proposal-lookup-service-86cf77cd87-tbqsv   5m           1178Mi
rabbitmq-0                                 241m         142Mi
request-consumables-697bd8cd5b-w7m7d       1m           82Mi
schedule-service-57dc7b5f89-jdk2w          2m           1963Mi
settings-tool-service-58fc879c7c-5tzmm     5m           590Mi
trino-d76869bb-wwhvt                       13m          3437Mi
user-office-web-service-55f8bf87c7-qlj7b   13m          1300Mi
user-wifi-login-service-6564fb8796-vpzn5   3m           726Mi
visits-service-75f95fc5f6-7ftql            5m           1373Mi

NAME                                       CPU(cores)   MEMORY(bytes)
approved-experiments-759cf565c4-cpn5t      0m           89Mi
auth-service-69fc9b6fc-fb8w2               1m           391Mi
clam-antivirus-5966d8867b-p749f            1m           1693Mi
cron-service-794c45ff7-fzxrr               1m           301Mi
gitea-d9bc6786d-s48r7                      2m           157Mi
mailchimp-webhooks-5bf5cb7459-wp8lt        1m           219Mi
merge-user-tool-6b49fdbb54-rl2tb           1m           315Mi
messages-service-8b9687ccb-l2sqg           2m           269Mi
panel-member-manager-775ddb9fbb-9bq4m      0m           64Mi
proposal-allocations-7b549b45bd-9657q      2m           760Mi
proposal-backend-54fc9877df-nbhcx          2m           312Mi
proposal-factory-cf5c678bd-pbggn           1m           294Mi
proposal-frontend-68c65d4749-dmncj         1m           10Mi
proposal-launchpad-69cccb94d8-sn2zd        0m           81Mi
proposal-lookup-service-86cf77cd87-tbqsv   5m           1405Mi
rabbitmq-0                                 246m         141Mi
request-consumables-697bd8cd5b-w7m7d       1m           82Mi
schedule-service-57dc7b5f89-jdk2w          2m           1964Mi
settings-tool-service-58fc879c7c-5tzmm     5m           591Mi
trino-d76869bb-wwhvt                       293m         4565Mi
user-office-web-service-55f8bf87c7-qlj7b   5m           1318Mi
user-wifi-login-service-6564fb8796-vpzn5   3m           726Mi
visits-service-75f95fc5f6-7ftql            9m           1373Mi