very slow when big search result

[steveyen]

on the intranet instance (running on a nice VM), search for "couchbase", which has over 82K hits. The response/result page is very slow.

Instead, getting the first page of incremental results quickly would be a good improvement.

Perhaps caching? Perhaps grab some first results, even if they're wrong (less relevant)? And, then, fixup/AJAX the the page when more relevant results show up?

[mschoch]

The problem is the way text search works there is no "incremental first page". In the worst case scenario every document we score has a higher score than the one before it. In which case we'd ultimately stream you every possible result. Plus, we'd have to do highlighting on each result, even though you'd only ever show the first page. Finally, the facet information wouldn't be available until we were done all results either.

[mschoch]

As a point of comparision, I loaded the intrapop bucket into Elasticsearch and ran the query:

{"query":{
    "match" : {
        "_all" : "couchbase"
    }
}}

Similar to searching for "couchbase" in intrapop. It took 277 ms. How is it so much faster? I don't know, but I do know that it scored all 82676 matching documents in that time.

[mschoch]

Also worth noting that this did not load any fields (couchbase+ES does not store them by default), did not do any highlighting, and did not compute any facets. This makes it not truly apples to apples comparison, but still useful datapoint.