Offer means to retrieve multiple primary names in a single request

[brent-hartwig]

Problem Description: LUX's design imposes a problematic ratio between document requests to other backend requests. This is known to have led to issues during the most recent test (#162) and will be a limiting factor when scaling concurrent production demand.

During #162, one of the application server's request queue was full for most of the test, causing the ML load balancer to return thousands of 504s per minute which, in turn, triggered the data service proxy's retry mechanism. The retry mechanism kept the test's error rate lower than it otherwise would have been but should not be relied upon to this extent and was not always successful. Limited to backend requests with a 200 status code, 57% (169,433 of 295,011) of the test's backend requests were document requests. 66% (112,384 of 169,433) of the document requests specified the name profile. The issue would have been exacerbated had the total number of backend requests been closer to what was expected (There were 55% to 70% fewer backend requests than expected.)

Outside the context of a performance, a single frontend search executed without any other concurrent demand was able to crash an instance of the v8 engine. That frontend search resulted in 197 backend requests:

• 1 search request
• 13 facet requests
• 20 document requests where profile=results: 1 per displayed search result. 182 of the 197 were for documents. 162 of the 182 were for primary names.
• 162 document requests where profile=name: the primary name of each facet value available before clicking a facet's "Show More" link.

While paginating facet requests (#160 and #161) now prevents the v8 engine crash, the crash required more than a single facet request. 91% of the requests on the node that experienced the v8 engine crash were document requests.

Expected Behavior/Solution: The first context I'd like to see fewer document requests in is search, which means facets. We can reduce from number of facets * up to 20 to number of facets * 1 or 2, depending on selected approach:

1. The most generic (i..e, more supported contexts) is to modify the document endpoint (or offer a new one) to accept multiple URIs. I would initially limit this ability to the name profile but any that return a relatively small amount of data should be fine. A maximum number of URIs would need to be imposed and should be no less than 20 (the maximum number of initially displayed facet values, per facet). In this case, there would be two backend requests to render a single facet, turning the 197 backend request example into 28 requests.
2. The targeted strike would be to modify the facet endpoint to include primary names for facets that have primary names. Now that we are committed to paginating facet values, we can include this amount of additional data without concern. The facet endpoint will need to offer the lang parameter and have the same default as the document endpoint. In this case, there would be one backend request to render a single facet, turning the 197 backend request example into 15 requests.

Requirements: Only bits to add to the above:

• [ ] Implement in a backwards-compatible manner, thereby allowing the backend change to rollout before the other tiers adopt and not imposing all endpoint consuming contexts to change at the same time.
• [ ] Update the API usage documentation

Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.

~- [ ] Wireframe/Mockup~

• [ ] Committee discussions - Sarah
• [ ] Feasibility/Team discussion - Sarah
• [x] Backend requirements
• [ ] Middle tier requirements: define in new middle tier ticket
• [ ] Frontend requirements: define in new frontend ticket
• [ ] Are new regression tests required for QA - Amy
• [ ] Compels a performance test?
• [ ] Questions
• List of questions for discussions. Answers should be documented within the issue.

UAT/LUX Examples: All frontend pages

Dependencies/Blocks: This ticket will block the associated frontend and middle tier tickets.

Related Github Issues: See the above Problem Description.

Related links: None.

Wireframe/Mockup: Not needed.

[brent-hartwig]

cc: @roamye, @clarkepeterf