Implement match_only_text handler for opensearch

[sandervandegeijn]

Is your feature request related to a problem? Please describe. I'm trying to use elastic common schema, it depends on the match_only_text handler:

https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html

Would be nice to use the template as is and to have the match_only_text either way

Describe the solution you'd like Implement match_only_text

Describe alternatives you've considered None

Additional context Add any other context or screenshots about the feature request here.

[macohen]

Thanks for opening this issue. Can you tell us more about your use case? It would be helpful to understand to target the solution.

[sandervandegeijn]

The opensearch security analytics plugins is dependant on the ECS schemas. So basically I wanted to load ECS in my opensearch cluster. It would seem that multiple handlers are currently missing causing import errors. Falling back on basic text fields does mitigate the issue, but it isn't good for interoperability.

[macohen]

Ah, so our dependency on ECS is localized to a plugin and that's problematic for consistent usage across the product. If we were to build ECS support into the base OpenSearch product, that would help solve your challenge. Tagging a few folks I think may be interested or able to help with this: @yang-db, @praveensameneni, @macrakis.

[YANG-DB]

Hi I think you are correct and the catalog repository is something that would be a fundamental part of this concept: https://github.com/opensearch-project/opensearch-catalog/

And specifically the observability simple scheme support...

https://github.com/opensearch-project/opensearch-catalog/tree/main/schema%2Fobservability

We are currently planning on providing an API to support these functionality and it may be reasonable that it will eventually be part of the core API

[sandervandegeijn]

Yip fully supporting ECS would help with the use of the security plugin, but since OS and ES are (still) very much related and ECS is kind of the standard it would be helpful to support it fully. It gives a common frame of reference when dealing with data structures and enables interoperability.

[msfroh]

@YANG-DB -- I'm going to assign this to you, since it feels most closely related to the work you're doing on simple schema support. Please feel free to unassign -- right now, with the "Search" label, it's showing up in our Search Relevance Triage queue.

[sandervandegeijn]

Great thanks, the ultimate check is to just load the whole set of ECS component templates into opensearch and see what breaks. There will be a couple of options missing.

This would greatly improve interoperability with ES :)

[macohen]

This looks like a potential performance improvement as well.

[msfroh]

I wonder if we could make match_only_text essentially an alias for text, but with index_options set to docs and norms set to false?

(I see that match_only_text doesn't say that it disallows queries that use positions, just that it performs more slowly on them. Maybe it does phrase/span queries by post-filtering against the source? Actually, the end of the documentation says that's what they do. Also, the end of that documentation says that a match_only_text field is essentially a text field with index_options set to docs and norms:false, so I guess I have the answer to my wondering above.)

[rishabhmaurya]

I'm picking this up. My plan is to run a benchmark with index_options set as docs and norms disabled. I will post the numbers soon.

[sandervandegeijn]

Cool, looking forward to it :)

[rishabhmaurya]

ok, so I have some initial numbers -

Environment setup

data node: 1
number of primaries: 5

Instance type: r5.2xlarge  
Number of vCPUs: 8
Heap configured: 32g

Benchmark client setup

Workload: pmc (https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/pmc)

To mimic the match_only_text field, I have modified the mapping for pmc index in the workload - index.json - modified the index_options and norms to -

"body": {
  "type": "text",
  "index_options": "docs",
  "norms": false
}

operations/default.json:

{
  "name": "query_string",
  "operation-type": "search",
  "#COMMENT": "Large responses cause overhead on the client when decompressing the response. Disable to avoid the overhead",
  "response-compression-enabled": false,
  "body": {
    "query": {
      "query_string": {
        "query": "body:(newspaper OR coverage)"
      }
    }
  }
}

Comparison

Storage

Using text field -

"primaries" : {
  "docs" : {
    "count" : 574199,
    "deleted" : 0
  },
  "store" : {
    "size_in_bytes" : 20974981986,
    "reserved_in_bytes" : 0
  }
}

Using mimic version of match_only_text field type for field body -

"primaries" : {
  "docs" : {
    "count" : 574199,
    "deleted" : 0
  },
  "store" : {
    "size_in_bytes" : 15024956253,
    "reserved_in_bytes" : 0
  }
}

15gb vs 20.9gb, so almost +25% storage efficient.

Indexing throughput

Text field -

|                                                  Segment count |                               |         125 |         |
|                                                 Min Throughput |                  index-append |     1116.98 |  docs/s |
|                                                Mean Throughput |                  index-append |     1158.05 |  docs/s |
|                                              Median Throughput |                  index-append |     1160.86 |  docs/s |
|                                                 Max Throughput |                  index-append |     1187.86 |  docs/s |
|                                        50th percentile latency |                  index-append |     2782.53 |      ms |
|                                        90th percentile latency |                  index-append |     4828.74 |      ms |
|                                        99th percentile latency |                  index-append |     11519.5 |      ms |
|                                       100th percentile latency |                  index-append |     15164.4 |      ms |
|                                   50th percentile service time |                  index-append |     2782.53 |      ms |
|                                   90th percentile service time |                  index-append |     4828.74 |      ms |
|                                   99th percentile service time |                  index-append |     11519.5 |      ms |
|                                  100th percentile service time |                  index-append |     15164.4 |      ms |

Mimic of match_only_text -

|                                                  Segment count |                               |         110 |         |
|                                                 Min Throughput |                  index-append |     1405.21 |  docs/s |
|                                                Mean Throughput |                  index-append |     1450.09 |  docs/s |
|                                              Median Throughput |                  index-append |     1455.06 |  docs/s |
|                                                 Max Throughput |                  index-append |     1474.86 |  docs/s |
|                                        50th percentile latency |                  index-append |     2189.04 |      ms |
|                                        90th percentile latency |                  index-append |     3211.79 |      ms |
|                                        99th percentile latency |                  index-append |     5114.12 |      ms |
|                                       100th percentile latency |                  index-append |      5658.7 |      ms |
|                                   50th percentile service time |                  index-append |     2189.04 |      ms |
|                                   90th percentile service time |                  index-append |     3211.79 |      ms |
|                                   99th percentile service time |                  index-append |     5114.12 |      ms |
|                                  100th percentile service time |                  index-append |      5658.7 |      ms |

increase in indexing median throughput from 1160.86 to 1455.06 ~25% increase and reduction in 90th percentile latency from 4828ms-> 3211.79ms ~33% reduction

Query string latency

With text

|                                                 Min Throughput |                          term |       19.94 |   ops/s |
|                                                Mean Throughput |                          term |       19.95 |   ops/s |
|                                              Median Throughput |                          term |       19.95 |   ops/s |
|                                                 Max Throughput |                          term |       19.96 |   ops/s |
|                                        50th percentile latency |                          term |     6.43933 |      ms |
|                                        90th percentile latency |                          term |     7.08113 |      ms |
|                                        99th percentile latency |                          term |     7.69279 |      ms |
|                                       100th percentile latency |                          term |     8.80695 |      ms |
|                                   50th percentile service time |                          term |     5.60147 |      ms |
|                                   90th percentile service time |                          term |     6.04863 |      ms |
|                                   99th percentile service time |                          term |     6.60193 |      ms |
|                                  100th percentile service time |                          term |      8.1905 |      ms |
|                                                     error rate |                          term |           0 |       % |
|                                                 Min Throughput |                        phrase |       19.86 |   ops/s |
|                                                Mean Throughput |                        phrase |       19.88 |   ops/s |
|                                              Median Throughput |                        phrase |       19.88 |   ops/s |
|                                                 Max Throughput |                        phrase |        19.9 |   ops/s |
|                                        50th percentile latency |                        phrase |     6.29159 |      ms |
|                                        90th percentile latency |                        phrase |     6.92876 |      ms |
|                                        99th percentile latency |                        phrase |     7.67065 |      ms |
|                                       100th percentile latency |                        phrase |     36.0957 |      ms |
|                                   50th percentile service time |                        phrase |     5.48194 |      ms |
|                                   90th percentile service time |                        phrase |     5.95093 |      ms |
|                                   99th percentile service time |                        phrase |     6.55707 |      ms |
|                                  100th percentile service time |                        phrase |      35.205 |      ms |
|                                                     error rate |                        phrase |           0 |       % |
|                                                 Min Throughput |                  query_string |       19.93 |   ops/s |
|                                                Mean Throughput |                  query_string |       19.94 |   ops/s |
|                                              Median Throughput |                  query_string |       19.94 |   ops/s |
|                                                 Max Throughput |                  query_string |       19.95 |   ops/s |
|                                        50th percentile latency |                  query_string |     8.14322 |      ms |
|                                        90th percentile latency |                  query_string |     8.89306 |      ms |
|                                        99th percentile latency |                  query_string |     9.88635 |      ms |
|                                       100th percentile latency |                  query_string |      10.122 |      ms |
|                                   50th percentile service time |                  query_string |     7.32504 |      ms |
|                                   90th percentile service time |                  query_string |      7.7771 |      ms |
|                                   99th percentile service time |                  query_string |     8.25705 |      ms |
|                                  100th percentile service time |                  query_string |     9.54034 |      ms |
|                                                     error rate |                  query_string |           0 |       % |

Mimic of match_only_text -

|                                                 Min Throughput |                          term |       19.96 |   ops/s |
|                                                Mean Throughput |                          term |       19.97 |   ops/s |
|                                              Median Throughput |                          term |       19.97 |   ops/s |
|                                                 Max Throughput |                          term |       19.97 |   ops/s |
|                                        50th percentile latency |                          term |     6.76727 |      ms |
|                                        90th percentile latency |                          term |      7.4203 |      ms |
|                                        99th percentile latency |                          term |     8.56366 |      ms |
|                                       100th percentile latency |                          term |     34.6451 |      ms |
|                                   50th percentile service time |                          term |     5.96028 |      ms |
|                                   90th percentile service time |                          term |     6.33442 |      ms |
|                                   99th percentile service time |                          term |     6.95427 |      ms |
|                                  100th percentile service time |                          term |     33.6502 |      ms |
|                                                     error rate |                          term |           0 |       % |
|                                        50th percentile latency |                        phrase |     2.82913 |      ms |
|                                        90th percentile latency |                        phrase |     3.12809 |      ms |
|                                        99th percentile latency |                        phrase |     4.41174 |      ms |
|                                       100th percentile latency |                        phrase |     5.20418 |      ms |
|                                   50th percentile service time |                        phrase |     2.82913 |      ms |
|                                   90th percentile service time |                        phrase |     3.12809 |      ms |
|                                   99th percentile service time |                        phrase |     4.41174 |      ms |
|                                  100th percentile service time |                        phrase |     5.20418 |      ms |
|                                                     error rate |                        phrase |         100 |       % |
|                                                 Min Throughput |                  query_string |       19.85 |   ops/s |
|                                                Mean Throughput |                  query_string |       19.87 |   ops/s |
|                                              Median Throughput |                  query_string |       19.87 |   ops/s |
|                                                 Max Throughput |                  query_string |       19.89 |   ops/s |
|                                        50th percentile latency |                  query_string |     8.93878 |      ms |
|                                        90th percentile latency |                  query_string |     10.0296 |      ms |
|                                        99th percentile latency |                  query_string |     10.6759 |      ms |
|                                       100th percentile latency |                  query_string |     11.9445 |      ms |
|                                   50th percentile service time |                  query_string |     7.96434 |      ms |
|                                   90th percentile service time |                  query_string |     8.37891 |      ms |
|                                   99th percentile service time |                  query_string |     8.86685 |      ms |
|                                  100th percentile service time |                  query_string |      10.641 |      ms |
|                                                     error rate |                  query_string |           0 |       % |

There is no improvement in query_string latency, it has slightly degraded!? or very close. Also, phrase queries are failing as you can see error is 100%

Conclusion

• significant gain (~25%) in index storage
• improvements in indexing - 25% increase in throughput and 33% reduction in indexing latency.
• no gains in query latency, number are very close.
• phrase query doesn't work out of the box and are resulting in errors.

Note: these are initial numbers and are published to understand the gains we might expect with this change.

cc: @msfroh @ict-one-nl @macohen

[rishabhmaurya]

I found this blogpost, thanks to Adrien for the writeup. It explains how elasticsearch solved it for phrase queries by the use of runtime fields and _source field. I will give it a shot and raise a PR soon using runtime fields to support phrase queries. Well, just found that we don't have support for the runtime fields and have an open RFC - https://github.com/opensearch-project/OpenSearch/issues/1133

[msfroh]

It explains how elasticsearch solved it for phrase queries by the use of runtime fields and _source field.

It sounds like it's more like it was solved using ideas from runtime fields. The fundamental idea is just that a phrase runs as a conjunction and then we can essentially post-filter by trying to match the phrase against the source (tokenizing on the fly?).

We might be able to do that with a custom Collector, for example. (On the other hand, if we can "easily" deliver runtime fields, that could be a nice double-win.)

[rishabhmaurya]

I have broken down the implementation of match_only_field into following -

1. New MatchOnlyTextFieldMapper and MatchOnlyTextFieldType with default FIELD_TYPE set as DOCS_ONLY and omit norms as true.
2. A new QueryBuilder and Query to filter the documents using _source field. Note: Same can be used for schema on reads https://github.com/opensearch-project/OpenSearch/issues/1133.
3. Since any positional queries doesn't work out of the box, so override of different match queries such as phrase and interval queries are overridden in MatchOnlyTextFieldType. The common part they are share would be - the individual token from the query string will be converted to Conjunction query.
4. Implementation of phrase query specific logic - The matches from conjunction query from 3 will be matched with query built using 2 where docs are matched further using _source field. The logic to match a document parsing field value using _source will be pluggable, so different query type can plug their logic.

I found lucene MemoryIndex (https://lucene.apache.org/core/8_0_0/memory/org/apache/lucene/index/memory/MemoryIndex.html) which I think could be pretty useful in implementing 4. Apparently all we need to do would be create single field single doc memory index and run the desired query against it (phrase/interval). (I felt too lazy to implement phrase query logic on a single doc myself and this seems like a lifesaver). Something like this -

        public Weight createWeight(IndexSearcher searcher, ScoreMode scoreMode, float boost) throws IOException {

                Weight weight = conjunctionQuery.createWeight(searcher, scoreMode, boost);

                return new ConstantScoreWeight(this, boost) {

                    @Override
                    public Scorer scorer(LeafReaderContext context) throws IOException {

                        Scorer scorer = weight.scorer(context);
                        DocIdSetIterator approximation = scorer.iterator();
                        LeafSearchLookup leafSearchLookup = lookup.getLeafSearchLookup(context);
                        TwoPhaseIterator twoPhase = new TwoPhaseIterator(approximation) {

                            @Override
                            public boolean matches() throws IOException {
                                leafSearchLookup.setDocument(approximation.docID());
                                List<Object> values = valueFetcher.fetchValues(leafSearchLookup.source());
                                MemoryIndex memoryIndex = new MemoryIndex();
                                for (Object value : values) {
                                    memoryIndex.addField(fieldType.name(), (String)value, fieldType.indexAnalyzer());
                                }
                                float score = memoryIndex.search(sourceQuery);
                                return score > 0.0f;
                            }

                            @Override
                            public float matchCost() {
                                // TODO: how can we compute this?
                                return 1000f;
                            }
                        };
                        return new ConstantScoreScorer(this, score(), scoreMode, twoPhase);
                    }

Let me know if this approach doesn't sounds reasonable? @msfroh

[harshavamsi]

Thanks @rishabhmaurya for breaking this down, I had some questions regarding 2 and 4:

• when you mean filter docs based on _source field, are you essentially talking about filtering out docs that only match terms in the phrase query? My assumption is that for phase queries with multiple terms we would want to tokenize each term in the query -> run a terms query on all those terms -> only run a _source phrase match on the docs that are remaining, I guess this way we are loading way lesser docs

• does memory index do a look up on _source?

[rishabhmaurya]

In this snippet, matches() is only called for docs matching individual terms of phrase query. Memory index will lookup into whatever field we index against it, in this case we are fetching the _source using ValueFetcher interface valueFetcher.fetchValues(leafSearchLookup.source()), which we are passing to memory index. I hope this clears your doubt.

[harshavamsi]

Yeah, this makes a lot of sense!

[sandervandegeijn]

Wow this has taken off, thanks guys :)

[msfroh]

I found lucene MemoryIndex (https://lucene.apache.org/core/8_0_0/memory/org/apache/lucene/index/memory/MemoryIndex.html) which I think could be pretty useful in implementing 4.

I think that would work.

You might also want to take a look at PercolateQueryBuilder, which creates in-memory indices to query them. It uses MemoryIndex if it needs to query a single document, but uses ByteBufferDirectory and an IndexWriter if it wants to process multiple documents.

On the other hand, I really like the self-contained matches() logic that you've got there. I don't see an obvious way to collect more than one candidate doc, since it's very likely that your phrase query would be part of a bigger conjunction calling advance on the approximations. At least you know that matches() is only going to get called once the top-level query has all the underlying approximate iterators pointing to the same doc. I feel like any attempt at batching in your custom scorer would end up indexing stuff that one of the other clauses would ignore anyway.

tl;dr: I thought of another way of doing it and talked myself out of it. I like your approach :+1:

[rishabhmaurya]

Here is the draft PR - https://github.com/opensearch-project/OpenSearch/pull/11039 Here are the pending items -

Testing

• [x] Unit tests
• [x] Test against span query and interval queries
• [x] Test with index phrases and prefix index settings
• [x] Test for multi match queries with blended fields
• [x] Test with gaps, positionIncAttr and slop settings for phrase and multiphrase queries
• [x] Test against suggest queries
• [x] Test expected behavior for all supported query types - behavior should match text field output.
• [x] Integ Tests

Documentation

• [ ] Create a table for all supported and unsupported operation.
• [ ] Document gains in index storage and latency one might expect.
• [ ] Document latency impact on phrase, multi-phrase and interval queries.
• [ ] Blog post

Benchmark

• [ ] Run benchmark using PMC workload and publish numbers and gains

PS: I'm currently oncall, I will resume on it later this week.

[msfroh]

@ict-one-nl -- can you please take a look at https://github.com/opensearch-project/OpenSearch/pull/11039 and let us know if the limitations are acceptable?

It sounds like we support phrase and multiphrase queries, but not e.g. interval and span queries for now.

[sandervandegeijn]

Yeah think that should be fine. It would be nice to maintain compatibility with Elastic where possible so ECS plus it's component templates work like expected in both opensearch and elasticsearch. This seems a valid tradeoff.

[rishabhmaurya]

Ran benchmark on PMC workload on same setup as before -

The phrase queries are now passing but with very high latency. Rest of the number are same as what we had for mimic version (text field with index_options set to docs) of match_only_text field before. I'm investigating further the reason for the high latency, they are expected to run slow and is the expected latency appropriate? It could also caused by large number of documents matching individual terms of the phrase query used in the benchmark https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/pmc/operations/default.json#L49 It could be because the TopFieldCollector might be collecting all the matching documents instead of cutting it short once it has collected the top N. I'm also going to run it against http_logs since that's where this field type is most applicable.

|                                                  Segment count |                               |         142 |         |
|                                                 Min Throughput |                  index-append |     1370.56 |  docs/s |
|                                                Mean Throughput |                  index-append |     1414.25 |  docs/s |
|                                              Median Throughput |                  index-append |      1416.9 |  docs/s |
|                                                 Max Throughput |                  index-append |     1427.01 |  docs/s |
|                                        50th percentile latency |                  index-append |     2122.88 |      ms |
|                                        90th percentile latency |                  index-append |     3206.27 |      ms |
|                                        99th percentile latency |                  index-append |     5615.57 |      ms |
|                                       100th percentile latency |                  index-append |     7325.38 |      ms |
|                                   50th percentile service time |                  index-append |     2122.88 |      ms |
|                                   90th percentile service time |                  index-append |     3206.27 |      ms |
|                                   99th percentile service time |                  index-append |     5615.57 |      ms |
|                                  100th percentile service time |                  index-append |     7325.38 |      ms |
|                                                     error rate |                  index-append |           0 |       % |
|                                                 Min Throughput |      wait-until-merges-finish |        0.03 |   ops/s |
|                                                Mean Throughput |      wait-until-merges-finish |        0.03 |   ops/s |
|                                              Median Throughput |      wait-until-merges-finish |        0.03 |   ops/s |
|                                                 Max Throughput |      wait-until-merges-finish |        0.03 |   ops/s |
|                                       100th percentile latency |      wait-until-merges-finish |     33382.3 |      ms |
|                                  100th percentile service time |      wait-until-merges-finish |     33382.3 |      ms |
|                                                     error rate |      wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                       default |       19.94 |   ops/s |
|                                                Mean Throughput |                       default |       19.95 |   ops/s |
|                                              Median Throughput |                       default |       19.95 |   ops/s |
|                                                 Max Throughput |                       default |       19.96 |   ops/s |
|                                        50th percentile latency |                       default |     5.91359 |      ms |
|                                        90th percentile latency |                       default |     6.51962 |      ms |
|                                        99th percentile latency |                       default |     7.29188 |      ms |
|                                       100th percentile latency |                       default |     41.0068 |      ms |
|                                   50th percentile service time |                       default |     5.05844 |      ms |
|                                   90th percentile service time |                       default |     5.47909 |      ms |
|                                   99th percentile service time |                       default |     6.76511 |      ms |
|                                  100th percentile service time |                       default |     40.3337 |      ms |
|                                                     error rate |                       default |           0 |       % |
|                                                 Min Throughput |                          term |       20.01 |   ops/s |
|                                                Mean Throughput |                          term |       20.01 |   ops/s |
|                                              Median Throughput |                          term |       20.01 |   ops/s |
|                                                 Max Throughput |                          term |       20.01 |   ops/s |
|                                        50th percentile latency |                          term |     6.43529 |      ms |
|                                        90th percentile latency |                          term |     7.04295 |      ms |
|                                        99th percentile latency |                          term |     7.62658 |      ms |
|                                       100th percentile latency |                          term |     8.34581 |      ms |
|                                   50th percentile service time |                          term |     5.61159 |      ms |
|                                   90th percentile service time |                          term |     6.09912 |      ms |
|                                   99th percentile service time |                          term |     6.64602 |      ms |
|                                  100th percentile service time |                          term |     8.00464 |      ms |
|                                                     error rate |                          term |           0 |       % |
|                                                 Min Throughput |                        phrase |        1.45 |   ops/s |
|                                                Mean Throughput |                        phrase |        1.46 |   ops/s |
|                                              Median Throughput |                        phrase |        1.46 |   ops/s |
|                                                 Max Throughput |                        phrase |        1.46 |   ops/s |
|                                        50th percentile latency |                        phrase |      381209 |      ms |
|                                        90th percentile latency |                        phrase |      433094 |      ms |
|                                        99th percentile latency |                        phrase |      444652 |      ms |
|                                       100th percentile latency |                        phrase |      445917 |      ms |
|                                   50th percentile service time |                        phrase |     692.869 |      ms |
|                                   90th percentile service time |                        phrase |     716.383 |      ms |
|                                   99th percentile service time |                        phrase |     729.919 |      ms |
|                                  100th percentile service time |                        phrase |     744.584 |      ms |
|                                                     error rate |                        phrase |           0 |       % |
|                                                 Min Throughput |                  query_string |          20 |   ops/s |
|                                                Mean Throughput |                  query_string |          20 |   ops/s |
|                                              Median Throughput |                  query_string |          20 |   ops/s |
|                                                 Max Throughput |                  query_string |          20 |   ops/s |
|                                        50th percentile latency |                  query_string |     9.40442 |      ms |
|                                        90th percentile latency |                  query_string |     10.4017 |      ms |
|                                        99th percentile latency |                  query_string |     10.9154 |      ms |
|                                       100th percentile latency |                  query_string |     11.4571 |      ms |
|                                   50th percentile service time |                  query_string |       8.227 |      ms |
|                                   90th percentile service time |                  query_string |      8.5564 |      ms |
|                                   99th percentile service time |                  query_string |      9.0237 |      ms |
|                                  100th percentile service time |                  query_string |     9.32245 |      ms |
|                                                     error rate |                  query_string |           0 |       % |
|                                                 Min Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                                Mean Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                              Median Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                                 Max Throughput | articles_monthly_agg_uncached |       19.97 |   ops/s |
|                                        50th percentile latency | articles_monthly_agg_uncached |      12.667 |      ms |
|                                        90th percentile latency | articles_monthly_agg_uncached |     13.1486 |      ms |
|                                        99th percentile latency | articles_monthly_agg_uncached |     13.5797 |      ms |
|                                       100th percentile latency | articles_monthly_agg_uncached |     14.7742 |      ms |
|                                   50th percentile service time | articles_monthly_agg_uncached |      11.787 |      ms |
|                                   90th percentile service time | articles_monthly_agg_uncached |      12.087 |      ms |
|                                   99th percentile service time | articles_monthly_agg_uncached |     12.3329 |      ms |
|                                  100th percentile service time | articles_monthly_agg_uncached |     13.7358 |      ms |
|                                                     error rate | articles_monthly_agg_uncached |           0 |       % |
|                                                 Min Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                                Mean Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                              Median Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                                 Max Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                        50th percentile latency |   articles_monthly_agg_cached |     4.81616 |      ms |
|                                        90th percentile latency |   articles_monthly_agg_cached |     5.23289 |      ms |
|                                        99th percentile latency |   articles_monthly_agg_cached |     13.5398 |      ms |
|                                       100th percentile latency |   articles_monthly_agg_cached |     34.3252 |      ms |
|                                   50th percentile service time |   articles_monthly_agg_cached |     3.96589 |      ms |
|                                   90th percentile service time |   articles_monthly_agg_cached |      4.2277 |      ms |
|                                   99th percentile service time |   articles_monthly_agg_cached |     13.1141 |      ms |
|                                  100th percentile service time |   articles_monthly_agg_cached |      33.706 |      ms |
|                                                     error rate |   articles_monthly_agg_cached |           0 |       % |
|                                                 Min Throughput |                        scroll |       12.55 | pages/s |
|                                                Mean Throughput |                        scroll |       12.59 | pages/s |
|                                              Median Throughput |                        scroll |       12.58 | pages/s |
|                                                 Max Throughput |                        scroll |       12.66 | pages/s |
|                                        50th percentile latency |                        scroll |     653.965 |      ms |
|                                        90th percentile latency |                        scroll |     695.542 |      ms |
|                                        99th percentile latency |                        scroll |     732.207 |      ms |
|                                       100th percentile latency |                        scroll |      732.32 |      ms |
|                                   50th percentile service time |                        scroll |     650.864 |      ms |
|                                   90th percentile service time |                        scroll |     692.698 |      ms |
|                                   99th percentile service time |                        scroll |     729.648 |      ms |
|                                  100th percentile service time |                        scroll |     729.885 |      ms |
|                                                     error rate |                        scroll |           0 |       % |

[rishabhmaurya]

It could be because the TopFieldCollector might be collecting all the matching documents instead of cutting it short once it has collected the top N.

that's causing the higher latency than expected for phrase queries. It tries to accurately track total hits upto 10000 hits as a default behavior set using track_total_hits. In this case its quite expensive as it would end up loading source for min(hits, track_total_hits). I'm disabling the track_total_hits and rerunning the pmc benchmark.

[rishabhmaurya]

the performance of phrase have improved significantly (reduced by half) on disabling track_total_hits.

|                                                 Min Throughput |                  index-append |     1355.31 |  docs/s |
|                                                Mean Throughput |                  index-append |     1395.68 |  docs/s |
|                                              Median Throughput |                  index-append |     1397.51 |  docs/s |
|                                                 Max Throughput |                  index-append |     1414.56 |  docs/s |
|                                        50th percentile latency |                  index-append |     1864.98 |      ms |
|                                        90th percentile latency |                  index-append |     3130.42 |      ms |
|                                        99th percentile latency |                  index-append |     5270.42 |      ms |
|                                       100th percentile latency |                  index-append |     6674.13 |      ms |
|                                   50th percentile service time |                  index-append |     1864.98 |      ms |
|                                   90th percentile service time |                  index-append |     3130.42 |      ms |
|                                   99th percentile service time |                  index-append |     5270.42 |      ms |
|                                  100th percentile service time |                  index-append |     6674.13 |      ms |
|                                                     error rate |                  index-append |           0 |       % |
|                                                 Min Throughput |      wait-until-merges-finish |        0.02 |   ops/s |
|                                                Mean Throughput |      wait-until-merges-finish |        0.02 |   ops/s |
|                                              Median Throughput |      wait-until-merges-finish |        0.02 |   ops/s |
|                                                 Max Throughput |      wait-until-merges-finish |        0.02 |   ops/s |
|                                       100th percentile latency |      wait-until-merges-finish |     63672.8 |      ms |
|                                  100th percentile service time |      wait-until-merges-finish |     63672.8 |      ms |
|                                                     error rate |      wait-until-merges-finish |           0 |       % |
|                                                 Min Throughput |                       default |       19.97 |   ops/s |
|                                                Mean Throughput |                       default |       19.98 |   ops/s |
|                                              Median Throughput |                       default |       19.98 |   ops/s |
|                                                 Max Throughput |                       default |       19.98 |   ops/s |
|                                        50th percentile latency |                       default |     6.79703 |      ms |
|                                        90th percentile latency |                       default |     7.37023 |      ms |
|                                        99th percentile latency |                       default |     8.57948 |      ms |
|                                       100th percentile latency |                       default |     90.3816 |      ms |
|                                   50th percentile service time |                       default |     5.81961 |      ms |
|                                   90th percentile service time |                       default |      6.1212 |      ms |
|                                   99th percentile service time |                       default |     6.88471 |      ms |
|                                  100th percentile service time |                       default |      89.416 |      ms |
|                                                     error rate |                       default |           0 |       % |
|                                                 Min Throughput |                          term |          20 |   ops/s |
|                                                Mean Throughput |                          term |          20 |   ops/s |
|                                              Median Throughput |                          term |          20 |   ops/s |
|                                                 Max Throughput |                          term |          20 |   ops/s |
|                                        50th percentile latency |                          term |     7.48985 |      ms |
|                                        90th percentile latency |                          term |     8.01574 |      ms |
|                                        99th percentile latency |                          term |     8.80889 |      ms |
|                                       100th percentile latency |                          term |     11.2657 |      ms |
|                                   50th percentile service time |                          term |     6.60418 |      ms |
|                                   90th percentile service time |                          term |     6.94315 |      ms |
|                                   99th percentile service time |                          term |      7.8463 |      ms |
|                                  100th percentile service time |                          term |     9.74326 |      ms |
|                                                     error rate |                          term |           0 |       % |
|                                                 Min Throughput |                        phrase |        3.29 |   ops/s |
|                                                Mean Throughput |                        phrase |        3.29 |   ops/s |
|                                              Median Throughput |                        phrase |        3.29 |   ops/s |
|                                                 Max Throughput |                        phrase |         3.3 |   ops/s |
|                                        50th percentile latency |                        phrase |      152127 |      ms |
|                                        90th percentile latency |                        phrase |      171719 |      ms |
|                                        99th percentile latency |                        phrase |      176182 |      ms |
|                                       100th percentile latency |                        phrase |      176704 |      ms |
|                                   50th percentile service time |                        phrase |     296.702 |      ms |
|                                   90th percentile service time |                        phrase |     317.196 |      ms |
|                                   99th percentile service time |                        phrase |     341.617 |      ms |
|                                  100th percentile service time |                        phrase |     344.592 |      ms |
|                                                     error rate |                        phrase |           0 |       % |
|                                                 Min Throughput |                  query_string |          20 |   ops/s |
|                                                Mean Throughput |                  query_string |          20 |   ops/s |
|                                              Median Throughput |                  query_string |          20 |   ops/s |
|                                                 Max Throughput |                  query_string |          20 |   ops/s |
|                                        50th percentile latency |                  query_string |     8.67978 |      ms |
|                                        90th percentile latency |                  query_string |     9.63559 |      ms |
|                                        99th percentile latency |                  query_string |     10.2941 |      ms |
|                                       100th percentile latency |                  query_string |     12.3806 |      ms |
|                                   50th percentile service time |                  query_string |     7.67736 |      ms |
|                                   90th percentile service time |                  query_string |     8.01091 |      ms |
|                                   99th percentile service time |                  query_string |     9.00617 |      ms |
|                                  100th percentile service time |                  query_string |     10.7269 |      ms |
|                                                     error rate |                  query_string |           0 |       % |
|                                                 Min Throughput | articles_monthly_agg_uncached |       19.95 |   ops/s |
|                                                Mean Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                              Median Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                                 Max Throughput | articles_monthly_agg_uncached |       19.96 |   ops/s |
|                                        50th percentile latency | articles_monthly_agg_uncached |     12.0627 |      ms |
|                                        90th percentile latency | articles_monthly_agg_uncached |     12.5037 |      ms |
|                                        99th percentile latency | articles_monthly_agg_uncached |     12.8665 |      ms |
|                                       100th percentile latency | articles_monthly_agg_uncached |     13.4782 |      ms |
|                                   50th percentile service time | articles_monthly_agg_uncached |     11.2122 |      ms |
|                                   90th percentile service time | articles_monthly_agg_uncached |     11.4519 |      ms |
|                                   99th percentile service time | articles_monthly_agg_uncached |     11.7431 |      ms |
|                                  100th percentile service time | articles_monthly_agg_uncached |     12.0608 |      ms |
|                                                     error rate | articles_monthly_agg_uncached |           0 |       % |
|                                                 Min Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                                Mean Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                              Median Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                                 Max Throughput |   articles_monthly_agg_cached |       20.02 |   ops/s |
|                                        50th percentile latency |   articles_monthly_agg_cached |     4.96532 |      ms |
|                                        90th percentile latency |   articles_monthly_agg_cached |     5.53372 |      ms |
|                                        99th percentile latency |   articles_monthly_agg_cached |     5.95282 |      ms |
|                                       100th percentile latency |   articles_monthly_agg_cached |     6.63634 |      ms |
|                                   50th percentile service time |   articles_monthly_agg_cached |     4.01379 |      ms |
|                                   90th percentile service time |   articles_monthly_agg_cached |     4.32317 |      ms |
|                                   99th percentile service time |   articles_monthly_agg_cached |     4.75762 |      ms |
|                                  100th percentile service time |   articles_monthly_agg_cached |     5.96884 |      ms |
|                                                     error rate |   articles_monthly_agg_cached |           0 |       % |
|                                                 Min Throughput |                        scroll |       12.55 | pages/s |
|                                                Mean Throughput |                        scroll |       12.58 | pages/s |
|                                              Median Throughput |                        scroll |       12.57 | pages/s |
|                                                 Max Throughput |                        scroll |       12.65 | pages/s |
|                                        50th percentile latency |                        scroll |     599.232 |      ms |
|                                        90th percentile latency |                        scroll |     622.499 |      ms |
|                                        99th percentile latency |                        scroll |     656.313 |      ms |
|                                       100th percentile latency |                        scroll |     686.954 |      ms |
|                                   50th percentile service time |                        scroll |     596.339 |      ms |
|                                   90th percentile service time |                        scroll |     619.489 |      ms |
|                                   99th percentile service time |                        scroll |     653.214 |      ms |
|                                  100th percentile service time |                        scroll |     683.882 |      ms |
|                                                     error rate |                        scroll |           0 |       % |

[rishabhmaurya]

Tasks -

• [x] https://github.com/opensearch-project/OpenSearch/pull/11039
• [x] https://github.com/opensearch-project/OpenSearch/pull/11714
• [x] https://github.com/opensearch-project/OpenSearch/pull/11736
• [x] https://github.com/opensearch-project/documentation-website/issues/5427
• [ ] https://github.com/opensearch-project/opensearch-benchmark-workloads/issues/151

[kiranprakash154]

Hi, are we on track for this to be released in 2.12 ?

[rishabhmaurya]

@kiranprakash154 yes this is on track for 2.12. Closing it.