Performance test: May 22, 2024 - Blue/TST: ML ver11.0.3

[jffcamp]

Primary objective: ~The purpose of this test depends on the results of the performance test run on May 21 #150. IF the May 21 performance test is successful, we will be deploying #100 and rerunning the test. This will be to understand any performance impact caused by expanding reference primary. IF the May 21 performance test shows early failures, we will be performing a fresh/clean install of ML 11.0.3 on Blue and will repeat the test~ Test run on May 21 was invalidated due to 10 flows not running. This was caused by a flow failing causing all following flows to fail. The objective is to replicate the results of Scenario J, using build 11.0.3, following yesterday's restart of the ML servers.

Code and Configuration Changes:

1. The related documents portion of the keyword search pattern switched from the referencePrimaryName field to the referenceName field. By search scope, here are the record types that are included or excluded from reference fields:
   • Included:
     • Agent: Group, Person
     • Concept: Language, MeasurementUnit, Type
     • Event: Activity, Period
     • Place: Place
   • Excluded:
     • Concept: Currency, Material
     • Item: DigitalObject, HumanMadeObject
     • Work: LinguisticObject, Set, VisualItem
2. All name search terms less the set search scope switched from their search scope-specific primary name field to their broader name field; for instance, the name search term in the agent scope was changed from agentPrimaryName to agentName.

Environment and versions: Blue (as TST) TBD comprised of MarkLogic 11.0.3 (downgraded from 11.2 early release), Backend v1.16.0, Middle Tier v1.1.9, Frontend v1.26, and Dataset produced on 2024-04-18.

Scenario AH of the Perf Test Line Up: our existing dual app server configuration (Scenario J) but with the above-discussed field difference. The last time Scenario J was tested is documented within https://git.yale.edu/lux-its/marklogic/issues/1033 (internal link).

Key metrics we're targeting (column E / scenario J):

Number of application servers: 2 per node. Maximum number of concurrent application server threads:

• lux-2: 12 per node for search and related list requests
• lux: 6 per node for all other request types
• total: 18 per node

For more information please see the documentation: LUX Performance Testing Procedure

Tasks to complete:

• [ ] QA to smoke test the performance scripts. Confirm all 74 flows run successfully
• ~[ ] Deploy Backend v1.16.0 with the fullTextSearchRelatedFieldName build property set to referenceName.~ Yesterday's test invalidated due to 10 flows missing. Rerunning yesterday's test w/out this change
• ~[ ] In QC, verify /lib/appConstants.mjs includes const FULL_TEXT_SEARCH_RELATED_FIELD_NAME = 'referenceName'.trim();~
• [x] Disable Blue's middle-tier caching.
• [x] Verify LUX trace events are enabled plus v8 delay timeout.
• [x] Verify no other v8-related trace events is enabled.
• [x] Smoke test the front end.
• [ ] Xinjian: Start collecting OS-level metrics.
• [x] Peter: Start collecting middle tier metrics (getMiddleTierStats.sh)
• [ ] QA: Verify/set ramp-up schedule to 2 simple search VUs, 1 filtered VU, and 1 entity page VU every three minutes until there are 148 users then hold for 15 minutes.
• [x] QA: Verify scripts point to BLUE/TST, https://lux-front-tst.collections.yale.edu/
• [ ] Team: Sign off on the above before proceeding.
• [ ] QA: Start performance test
• [ ] Team: check total request count at 10 minutes
• [ ] Team: check total request count at 15 minutes
• [ ] Team: check total request count at 20 minutes
• [ ] QA: Finish performance test

Data collection (Details from procedure):

• [x] Xinjian: Stop collecting OS-level metrics and attach to the ticket
• [x] Peter: Stop collecting middle-tier metrics and attach to the ticket
• [ ] Xinjian: Collect data from AWS and attach to ticket.
• [ ] TBD: Download the monitoring history (level=raw) and attach to the ticket.
• [ ] TBD: Take screenshots of select monitoring history graphs.
• [ ] TBD: Collect, trim, and attach backend logs to the ticket.
• [ ] TBD: Pull app server queue metrics, attach to the ticket, and record in Perf: Key Metrics.
• [ ] TBD: Update online spreadsheet tabs with what is known at this point.

Revert all configuration changes:

• [ ] Deploy Backend v1.16.0 with the fullTextSearchRelatedFieldName build property set to referencePrimaryName.
• [x] Enable middle-tier caching.

Verify:

• [x] Smoke test the front end.

Analysis:

• [ ] TBD: Upon receipt, review report from QA and update related portions of the online spreadsheet tabs.
• [ ] TBD: Mine the backend logs?
• [ ] TBD: Determine if the test is valid.
• [ ] TBD: Determine if the performance is acceptable --> #100.

[jffcamp]

Flows were updated to increase the wait from 3 to 5 seconds for page load. QA's smoke test was triggering v8 crashes. Test stopped.

[roamye]

Approved by UAT

[roamye]

Closing as this ticket was marked as Done the week of 6/3.