brent-hartwig
commented
1 month ago This ticket's scope may be too broad. It's typically better to focus on one timeout / perf issue at a time.
In the case of requesting the last page of a search's results, the frontend is specifying pagination parameters that has the current implementation of the backend perform a search that has to filter results through page * page length. Filtering is the process of pulling the document from disk and through d-node then e-node caches to validate it meets all search criteria. The process may end up dropping some --when indexes alone were unable to apply all of the criteria (unfiltered).
When searching for objects containing "colors", the estimate (unfiltered) comes back 109,127. With cold caches, it took 22.5 seconds to get to the last seven. With warm caches, it still took 16.1 seconds. An unfiltered search consistently returns the same last seven results in 117 milliseconds, illustrating this search can be accurately resolved by indexes alone and that the filtering process is adding 16 or seconds to double check the 109K results.
Things we may want to discuss:
- Switching to unfiltered searches and seeking out instances where false positives show up. On this project, there have been legitimate and illegitimate instances attributed to differences between unfiltered and filtered results. It would be helpful to isolate the legitimate instances and split that list between those that can be eliminated via index changes and those that will persist --then decide if we can switch to unfiltered searches and reap the performance benefits.
- When the requested page is more than 50% into cts.estimate, reverse the sort and paginate/filter the other way. For more details and ideas, see https://git.yale.edu/lux-its/marklogic/issues/1010 (internal).
- Do not offer a Last page link in the frontend. What value does it offer? Users can reverse the sort when interested in those results.
- Is it reasonable/expected for a relatively quiet cluster of three 16 vCPU 128 GB memory nodes to take 16 seconds to filter 109K results? Below is this search's generated CTS query, specifying what the filtering process would have had to validate in each unfiltered result. I posted an internal inquiry.
cts.andQuery([
cts.jsonPropertyValueQuery(
'dataType',
['DigitalObject', 'HumanMadeObject'],
['exact']
),
cts.orQuery([
cts.fieldWordQuery(
['itemAnyText'],
'colors',
[
'case-insensitive',
'diacritic-insensitive',
'punctuation-insensitive',
'whitespace-insensitive',
'stemmed',
'wildcarded',
],
1
),
cts.tripleRangeQuery(
[],
[[lux('itemAny')]],
fn.insertBefore(
cts.values(
cts.iriReference(),
'',
['eager', 'concurrent'],
cts.fieldWordQuery(
['referencePrimaryName'],
'colors',
[
'case-insensitive',
'diacritic-insensitive',
'punctuation-insensitive',
'whitespace-insensitive',
'stemmed',
'wildcarded',
],
1
)
),
0,
sem.iri('/does/not/exist')
),
'=',
[],
1
),
]),
]);
clarkepeterf
Problem Description: There have been several time out errors in Advanced Search queries which only have one multiple field. (see example below) This issue serves as a research ticket for all performance enhancements for time out issues not related to #113 to figure out how to solve this issue.
Another time out issue is when users select the last page of a search result. (example below)
Expected Behavior/Solution: Research on how to fix this time out issue. Solution is TBD Possible solution: increase our time out
Requirements: TBD
Needed for promotion: If an item on the list is not needed, it should be crossed off but not removed.
UAT/LUX Examples:
Dependencies/Blocks:
Related Github Issues:
Related links:
Wireframe/Mockup: Place wireframe/mockup for the proposed solution at end of ticket.