Performance Test - scheduled for 2024-07-09

jffcamp commented 1 month ago

Primary Objective

Use Scenario J Determine if there is an adverse impact on performance with the change to Timeline pagination and returning up to 10,000 results per page. This is to be compared against the new baseline confirmed against ML 11.0.3. This will be a 'lightweight' test. We will assemble at the start, make sure the test begins well, check for crashes for the first 10 minutes. When the test concludes, collect and grep logs for crashes.. Once the QA report is available, we can complete the performance tabs on the Environment spreadsheet.

Changes Being Tested

After the regression test has been completed, deploy the code that includes timeline pagination (if it isn't already part of 252) https://github.com/project-lux/lux-frontend/issues/252

Context

We have confirmed good performance against ML 11.0.3 using the new facet pagination. We have NOT tested performance with the expansion of the timeline/date pagination for entity pages. This will be tested against a previously validated system, before introducing the upgrade to ML 11.2 or 11.3.

Environment and Versions

Environment: TST which is configured to the Green backend.
MarkLogic 11.0.3
Backend 1.20.0
Middle Tier 1.11.12
Frontend v1.29.3 with timeline pagination
Dataset produced on 2024-05-29

Backend Application Server Configuration

lux-request-group-1 on port 8003: The middle tier is expected to send all requests here except search and relatedList requests. Maximum of 6 concurrent requests.
lux-request-group-2 on port 8004: The middle tier is expected to send all search and relatedList requests to this application server. Maximum of 12 concurrent requests.
Maximum of 18 concurrent requests per node.

Tasks

For more information please see the documentation: LUX Performance Testing Procedure

Prep, Start, and Preliminary Checks

[x] Confirm the most recent blue-green switch is 100% complete (i.e., no part of TST is using PROD).
[x] Deploy code and/or configuration changes that are being tested.
[x] Disable middle-tier caching in TST (instructions).
[x] Verify LUX trace events are enabled plus v8 delay timeout.
[x] Smoke test the front end.
[x] Start collecting OS-level metrics (instructions). ~- [ ] Start collecting middle tier metrics (script)~
[x] QA: Verify/set ramp-up schedule to 2 simple search VUs, 1 filtered VU, and 1 entity page VU every three minutes until there are 148 users then hold for 15 minutes.
[x] QA: Verify test is configured with three second wait times.
[x] QA: Verify scripts point to TST, https://lux-front-tst.collections.yale.edu/
[x] Team: Sign off on the above before proceeding.
[x] QA: Start performance test. 12:57 PM EDT
[x] Team: begin monitoring for v8 engine crashes.
[ ] Team: check total request count at 10 minutes.
[ ] Team: check total request count at 15 minutes.
[x] Team: check total request count at 20 minutes.
[ ] QA: Finish performance test.

Collect data

[ ] Stop collecting OS-level metrics and attach to the ticket.
[ ] Stop collecting middle-tier metrics and attach to the ticket.
[ ] Collect data from AWS and attach to ticket.
[ ] Collect ML monitoring history (instructions).
[ ] Collect (script), trim (script), and attach backend logs to the ticket.
[ ] Pull app server queue metrics (script), attach to the ticket, and record in Perf: Key Metrics. Starting in ML 11.2.0, this information may be included in the ML monitoring history screenshots and exports.
[ ] Update online spreadsheet tabs with what is known at this point.

Restore and Verify Environment

[ ] Revert this test's code and configuration changes
[ ] Enable middle-tier caching (instructions).
[ ] Smoke test the front end.

Analyze

[ ] Upon receipt, review report from QA and update related portions of the online spreadsheet tabs.
[ ] Mine the backend logs?
[ ] Determine if the test is valid.
[ ] Determine if the performance is acceptable.