incorrect Result Count for large results

NCIOCPL / sitewide-search-api

0 stars 4 forks source link

incorrect Result Count for large results #83

Closed zhuomingao closed 2 years ago

zhuomingao commented 2 years ago

Issue description

searches that return > 10000 results, always has result counts 10000 returned from ES query

solution

Add "track_total_hits": true to ES query

blairlearn commented 2 years ago

Two search terms, reported by @zhuomingao:

one thing I notice is "totalResults" is always 10000, for keywords like cancer, breast cancer, e.g. curl -X 'GET' \ 'https://webapis-dev.cancer.gov/sitewidesearch/v1/Search/cgov/en/cancer?from=0&size=10&site=all' \ -H 'accept: application/json'

We'll want to add integration tests for those.

blairlearn commented 2 years ago

Our data set isn't large enough to produce a totalResults value greater than 10,000. The new plan then is

Add the track_total_hits clause to the ES query.
Test on webapis-dev to make sure it works there.
Add unit tests to assure the track_total_hits clause isn't dropped from the query.