albertisfu
commented
9 months ago Here are the assets used for:
Profiling:
profile_all_child_all_hl_plain_999_limit.json profile_all_child_hl_except_plain.json profile_all_child_hl_only_plain.json profile_all_child_no_hl.json profile_all_child_no_hl_no_exclude_plain.json profile_no_child.json profile_normal_all_child_all_hl.json
Benchmarking:
Text query: track_all_child_all_hl_plain_999_limit.json track_all_child_hl_only_plain.json track_all_child_no_hl.json track_no_child.json track_normal_all_child_all_hl.json track_all_child_hl_except_plain.json
Number of child documents (inner_hits) displayed
1_child_all_hl.json 2_child_all_hl.json 3_child_all_hl.json 4_child_all_hl.json 5_child_all_hl.json 6_child_all_hl.json
Type of query: track_exact_text_query.json track_filter_case_name_text_query.json track_filter_case_name.json track_match_all.json track_normal_text_query.json
mlissner
@mlissner As we talked about, I've been investigating this issue to determine what is causing some queries to be too slow. Spoiler: highlighting the
plain_textfield is the main reason.Query Profiling I started by profiling a text query for RECAPSearch.
I populated my testing cluster with 2,079 cases and 15,936 entries, all of them took from the dev database, so it's real data.
To profile a query it's only required to add
"profile":trueto the query request body and set the cache tofalse.And it will return the normal results with an additional profile key. Here is an example: profile_response.txt
The profiled query was a simple text query:
?type=r&q=apple&order_by=score%20descThese are the results.
Normal, all child, all HL This is the query currently being used in production; it includes up to 6 child documents with the following highlighted fields for each child doc:
In this graph showing the results, it is possible to observe the time taken for each query component, measured in ms:
From this, we can see that the phase taking the most time is
InnerHitsPhase.Adding up all the phases, we get a total of
235.908916 msfor the entire request.No child at all Then I performed the same query excluding all the child documents (inner_hits) from the results:
Total query time:
2.963787 msFrom this, we can confirm that the phase taking the most time is the display of child documents.All child, no HL In the following query, I tested keeping child documents but removing the HL of child fields.
Total query time:
10.64146 msFrom here, we can see that including child documents in the results without applying highlighting increases the query time, but not as significantly as the query that includes HL.
So, the main problem appears to be the highlighting of child documents.
All child, excluding plain_text HL The following query includes child documents and HL for all the child fields except for
plain_textTotal query time:
14.377129 msHere we can see that HL of
plain_textis taking the most time.All child, only plain_text HL And we can confirm that by checking the inverse query, including only the
plain_textHL.Total query time:
219.818077 msFrom the following query, we can confirm that the issue is not related to extracting
plain_textand sending it in the results. This is considering that our query excludesplain_textfrom_sourceand instead always displays aplain_textfragment via highlighting when the field contains content. All childs, no HL, avoid excluding plaint_textTotal query time:
8.356079 msFinally from the following query profile we can confirm the issue is the HL of of
plain_textand is directly related to the amount of content to be HL. By default our current HL process until999_999chars which is the max allowed. Reducing it to999, decreased the query time.All childs, All HL, limit plain_text HL to 999
Total query time:
18.753498 msWe can observe that there is the
HighlightPhase, which doesn't take much time, and also theInnerHitsPhase. According to these tests, it seems that theInnerHitsPhaseperforms the highlighting for child fields and also processes the extraction of child documents and their fields to render them into the results.In the following graph, we can clearly observe all the previous queries and their respective times:
Benchmarking
The query times shown in the previous profiles vary with each request. I tried to take an average value, but they're not statistically representative. Consequently, I decided to also perform benchmarking to obtain more representative values, including the variable of load. We can replicate this benchmarking in our production cluster to have a reference before launching into production.
The benchmarks were conducted on my local testing cluster, which contains the same number of documents as reported in the previous profiles.
Using a current cluster with documents ensures that results are more accurate compared to an isolated environment with limited documents and different resources from the production cluster.
If we run these benchmarks in the production cluster, it is advisable to do so before going live, as the benchmarks are resource-intensive. It's also recommended to ensure snapshots are up-to-date as a precaution in case something goes wrong.
We'll need a pod that has connectivity to the production cluster via the load balancer. And install rally.
pip3 install esrallyThen we need to place the race assets in a directory. (I'll attach all the assets in a following comment.)
And then run:
esrally race --pipeline=benchmark-only --track-path=tracks/recap --target-hosts=localhost:9200 --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'password'"After tweaking the client options.
Here are the results I got from these benchmarks.
The benchmarking requests have the following parameters:
6 child, all HL This is the query that is currently being used in production, it includes up to 6 child documents with all the selected fields HL.
6 child, No HL All the child documents but not HL.
6 child, HL only
plain_text6 child, all HL except `plain_text
No child Query that doesn't display
inner_hitsat all.6 Childs, all HL, limit
plain_textHL to 999 chars Query with all the child fields HL but limitingplain_textHL to 999 chars.In the following graph we can see these results more clearly.
I also conducted some additional benchmarks to evaluate other variables.
Like the number of child documents (inner_hits) displayed.
From here, we can see that with all the HL activated, we achieve the best performance by retrieving only 1 child document. From 2 child documents onwards, performance starts to decrease.
Or the type of query requested.
From here, we can see that performance also depends on the number of documents matched and the type of query requested. For instance, a
Match Allquery has better performance than a text query with fewer documents matched, because theMatch Allquery does not perform highlighting.A filter or a filter combined with a text query can have good performance as long as only a few documents are matched.
In brief:
From these benchmarks and the previous profiles, we can conclude that the process taking the most time is the highlighting of
plain_text. The worst performance is observed in our current query, which includes 6 child documents and highlights all selected child fields. You can notice that best performance is achieved with a query that does not display child documents. However, since displaying child documents is mandatory for our use case, our optimization goal could be to achieve a performance similar to the query '6 children, all HL except plain_text'. Therefore, we should consider the performance when assessing other highlighters in #3300 to see if it's possible to gain additional efficiency in this process.