Investigate failures experienced during performance/stress test (from 965)

[gigamorph]

Impacted MarkLogic processes auto recover, holding up the other nodes for approximately 20 seconds. Multiple retry mechanisms reduce the likelihood of failed requests. Does not significantly increase the performance test error rate. Not a current production issue.

In Feb 2024, we ran the performance test against an 11.2 nightly build and were able to reproduce this issue. Logs and other artifacts were provided to Support via support ticket no. 35746. Our ticket: #1165.

Problem Description: During all recent performance tests, the v8 engine crashed multiple times on every node. James Kerr, of MarkLogic Engineering, requests that we report all v8 engine crashes regardless of context. This could be very beneficial as they may figure out one or more issues while we continue our investigation in parallel .

This ticket is for us to open such a support ticket and serve up the information enabling them to look into the crashes.

James confirmed that a v8 engine crash leads to a MarkLogic segfaults; although, let's keep an eye out for unrelated segfaults.

Expected Behavior/Solution: TBD

Requirements:

• 150 concurrent users to hit the system and for ML to be performant and not crash
• TBD

Needed for promotion: TBD

If an item on the list is not needed, it should be crossed off but not removed.

• [ ] ~Wireframe/Mockup - Heather~
• [ ] ~Committee discussions - Sarah~
• [ ] Feasibility/Team discussion - Sarah
• [ ] Backend requirements - TBD
• [ ] ~Frontend requirements- TBD~
• [x] Questions
• List of questions for discussions. Answers should be documented within the issue.

UAT/LUX Examples: TBD

Dependencies/Blocks: TBD

Related Github Issues:

• https://git.yale.edu/lux-its/marklogic/issues/560 (closed)
• https://git.yale.edu/lux-its/marklogic/issues/1024 (open)
• https://git.yale.edu/lux-its/marklogic/issues/927 (closed)

Related links:

• https://help.marklogic.com/Tickets/Ticket/View/35746 (Mark Logic Support Ticket)

Wireframe/Mockup: TBD

cc: @jac237, @prowns, @guoxinji

[brent-hartwig]

@jffcamp, @prowns, @xinjianguo, @clarkepeterf,

ML Engineering has paused their efforts on investigating the instance we submitted from Feb 2024's performance test. They stand a better chance at tracking down the cause in more controlled or isolated conditions. It is not practical for Engineering to replicate the LoadRunner performance test; thus, we're going for more isolated conditions, which is where the new monitoring test for messages containing "hung" come in. Should that test alert, let's learn what we can about the conditions and pass on the logs via Support ticket no. 35746. Second, next time we run the performance test, we should consider doing so without the custom error handler (#89).

[brent-hartwig]

Additional requests from ML Engineering for the next performance test:

• Enable the “v8 delay timeout” event trace
• Collect all logs, configurations, and related information for the app server and system environment which should include but not limited to:
  • Which event traces are enabled.
  • Timeout limits.
  • Error handler setting.
  • Latest version of the custom error handler, if used.