Nodejs Apps Memory Adjustments

andrewsignori-aot commented 1 month ago

The SIMS Api demonstrated some degradation on TEST where a preliminary analysis indicated that it may be caused by memory being consumed and not well handled leading the POD to respond with a major delay causing timeouts in the Web application request (e.g. related ticket #3224).

Technical

[ ] Increase the memory allocated to SIMS-Api, queue-consumers, and workers.
- Suggested
- MEMORY_LIMIT: from 256M to 512M
- MEMORY_REQUEST: from 128M to 384M
[ ] Edit the HorizontalPodAutoscaler memory for SIMS-Api, queue-consumers, and workers.
- Update AverageValue from 230M to 350M
[ ] Add the --max-old-space-size=250 to all prod scripts.
- "start:prod:api": "node --enable-source-maps dist/apps/api/main.js --max-old-space-size=250"
- "start:prod:workers": "node --enable-source-maps dist/apps/workers/main.js --max-old-space-size=250",
- "start:prod:queue-consumers": "node --enable-source-maps dist/apps/queue-consumers/main.js --max-old-space-size=250",

Additional context

Current memory consumption on PROD on May 15th, 2024.

limits: 26.8%
requests: 18.9%

Current SIMS-Api memory on PROD on May 15th, 2024.

andrewsignori-aot commented 1 month ago

Below is an article with some additional information on the default Nodejs max memory.

The same article has the suggested fix.

Source: https://medium.com/geekculture/node-js-default-memory-settings-3c0fe8a9ba1

Other references:

andrewsignori-aot commented 3 weeks ago

Load test result

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: ./dist/institution-get-profile.test.js
     output: -

  scenarios: (100.00%) 1 scenario, 240 max VUs, 11m0s max duration (incl. graceful stop):
           * default: Up to 240 looping VUs for 10m30s over 2 stages (gracefulRampDown: 30s, gracefulStop: 30s)

     ✓ Retrieved with success

     █ setup

     checks.........................: 100.00% ✓ 129514     ✗ 0
     data_received..................: 281 MB  445 kB/s
     data_sent......................: 217 MB  343 kB/s
     http_req_blocked...............: avg=3.01ms   min=0s      med=0s      max=1.04s    p(90)=0s       p(95)=33.35ms
     http_req_connecting............: avg=1.3ms    min=0s      med=0s      max=1.03s    p(90)=0s       p(95)=14.4ms
     http_req_duration..............: avg=132.35ms min=15.68ms med=25.69ms max=3.49s    p(90)=223.76ms p(95)=892.07ms
       { expected_response:true }...: avg=132.35ms min=15.68ms med=25.69ms max=3.49s    p(90)=223.76ms p(95)=892.07ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 130707
     http_req_receiving.............: avg=118.88µs min=0s      med=0s      max=7.52ms   p(90)=530.9µs  p(95)=807.73µs
     http_req_sending...............: avg=9.19µs   min=0s      med=0s      max=51.71ms  p(90)=0s       p(95)=0s
     http_req_tls_handshaking.......: avg=1.7ms    min=0s      med=0s      max=346.35ms p(90)=0s       p(95)=18.28ms
     http_req_waiting...............: avg=132.22ms min=15.46ms med=25.56ms max=3.49s    p(90)=223.24ms p(95)=892.04ms
     http_reqs......................: 130707  207.101887/s
     iteration_duration.............: avg=1.14s    min=0s      med=1.03s   max=4.63s    p(90)=1.26s    p(95)=1.89s
     iterations.....................: 129514  205.211609/s
     vus............................: 16      min=8        max=240
     vus_max........................: 240     min=240      max=240

running (10m31.1s), 000/240 VUs, 129514 complete and 0 interrupted iterations
default ✓ [======================================] 000/240 VUs  10m30s

API CPU/Memory during load test

andrewsignori-aot commented 1 week ago

Openshift Tests

Parallelism allowed for bulk insert: 1 Liveness Probe: 3000 Readiness Probed: 1000 CPU Limit: 0.2 CPU Requested: 0.1 Memory Limit: 768M Memory Requested: 512M max-old-space-size=550 Process started at 12:46:53 PM Finished at 12:53:47 PM (7 min)

andrewsignori-aot commented 1 week ago

Openshift Tests

Parallelism allowed for bulk insert: 4 Liveness Probe: 3000 Readiness Probed: 1000 CPU Limit: 0.2 CPU Requested: 0.1 Memory Limit: 768M Memory Requested: 512M max-old-space-size=550 Process started at 2:16:07 PM Finished at 2:22:06 PM (6 min)

andrewsignori-aot commented 1 week ago

Openshift Tests

Parallelism allowed for bulk insert: 4 Liveness Probe: 3000 Readiness Probed: 1000 CPU Limit: 0.3 CPU Requested: 0.2 Memory Limit: 1024M Memory Requested: 768M max-old-space-size=700 Process started at 2:51:19 PM Finished at 2:56:07 PM (5min)

andrewsignori-aot commented 1 week ago

Openshift Tests

Parallelism allowed for bulk insert: 4 Liveness Probe: 3000 Readiness Probed: 2500 CPU Limit: 0.2 CPU Requested: 0.1 Memory Limit: 768M Memory Requested: 512M max-old-space-size=500 Process started at 4:40:18 PM Finished at 4:46:41 PM (6min) maxStalledCount: 0, lockDuration: 60000, lockRenewTime: 5000,

bcgov / SIMS