Paginate non-semantic facet requests

[brent-hartwig]

Problem Description: Each facet request returns all of the facet's values, leading to requests taking longer than necessary to process, response sizes being larger than necessary, and contributing to system instability. This ticket's scope is limited to non-semantic facets.

Expected Behavior/Solution: Paginate facet requests. Default to 20 facet values. Impose a maximum of 100 values per request.

Until the frontend and middle tier implement facet pagination, the frontend will be limited to the first page of facets, as defined by the backend.

Requirements: Enable backend endpoint consumers to specify the page and page length. Default to page 1 with a page length of 20.

Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.

- [ ] Wireframe/Mockup - Mike - [ ] Committee discussions - Sarah - [ ] Feasibility/Team discussion - Sarah

• [x] Backend requirements: See above
• [x] Middle tier requirements: https://github.com/project-lux/lux-middletier/issues/65
• [x] Frontend requirements: https://github.com/project-lux/lux-frontend/issues/252
• [ ] Are new regression tests required for QA - Amy

- [ ] Questions - List of questions for discussions. Answers should be documented within the issue.

UAT/LUX Examples:

• Example endpoints from LUX to use for development and testing, if applicable.

Dependencies/Blocks:

• Blocked By: Nothing
• Blocking: The associated frontend and middle tier tickets.

Related Github Issues:

161: Same ticket but for semantic facets.

162: Performance test targeting #160 if not also #161

Related links:

None

Wireframe/Mockup: Place wireframe/mockup for the proposed solution at end of ticket.

[brent-hartwig]

First pass implemented in https://github.com/project-lux/lux-marklogic/commit/49e6013ee02c349e0f120224339d3cfb90038908 of branch https://github.com/project-lux/lux-marklogic/commits/160-paginate-non-semantic-facet-requests/. Tested well in DEV. Using the now infamous search where scope=work and criteria={"createdBy":{"AND":[{"gender":{"name":"female"}},{"startAt":{"classification":{"name":"city"}}}]}}:

• Asking for page 1, pageLength 20 of the workAboutConceptId facet took between 993 and 1,083 ms when hitting the backend endpoint directly.
• Asking for page 14,000, pageLength 20 of the same facet took 1,015 and 1,045 ms.
• Below are the response times for all facets from the same search, when executed in the frontend.

I did not attempt to optimize the code that formats the facet values as Activity Streams. That can be the scope of a different ticket, especially since multiple endpoints use that code.