ES caps total hits at 10K?

[philbudne]

It looks like https://github.com/mediacloud/story-indexer/issues/166 is an ES query API issue, so opening issue here.

[philbudne]

Summary: To get real total hits, you now need to set a (possibly expensive) parameter track_total_hits:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#track-total-hits

@rahulbot wrote:

Ugh. That's ridiculous (from a database user perspective). This is critical data we need to show users on every search result. Could we: a. sum the attention-over-time data and use that to provide an estimated total? (Django front-end server could do this, or mediacloud-news-search library used by mc-providers) b. turn on the expensive solution in staging and measure true impact on our data?

[philbudne]

The "b." alternative begs a question: how best to develop/test news-search-api, which I've created this issue for: https://github.com/mediacloud/news-search-api/issues/27

[pgulley]

Sorry- I missed this thread. I've just gone ahead and added the track_total_hits parameter to the overview query. I'll ping IA tomorrow to see how they addressed this issue in their system, but I figure we can play with this for now.

[rahulbot]

I'm getting real numbers back in the providers api now (ie. bigger than 10,000) 🎉 Is this fix deployed?

[pgulley]

Yes! Closing