Search response returns incorrect number of documents

[akolhun]

Describe the bug Search response returns root.fields.totalCount = X, but in fact lesser number of documents is returned

To Reproduce Steps to reproduce the behavior:

1. launch vespa cluster with 12 pods (helm chart attached): helm create vespa -n vespa-privatemedia --create-namespace .
2. load vespa application package (attached)
3. load attached data.json via vespa-feeder cli tool: vespa-feeder data.json
4. execute a query as:

   curl 'http://localhost:8080/search/' \
   --header 'Content-Type: application/json' \
   --data '{
   "yql": "select * from mp_private_media where site_id contains '\''c5402062-bedf-4e3e-80ad-d668993ed9b2'\'' and state contains '\''trash'\''",
   "hits": 100,
   "offset": 0
   }'

   Response contains root.fields.totalCount=54, but in fact 38 docs get returned

Expected behavior Response should contain 54 docs, as root.fields.totalCount claims

Environment (please complete the following information):

• OS: Amazon Linux
• Infrastructure: Kubernetes
• Versions: v1.24.17-eks

Vespa version 8.408.12

Additional context Note: the problem varies based on the number of nodes defined in content cluster. Looks like it's a distribution key releated issue

vap_privatemedia.zip vespa_privatemedia_helm.zip data.json.zip

[hmusum]

This could be due to timeout, see the doc on timeout and the further documentation this points to, e.g. soft timeout

See also documentation about summaries, especially the section on performance

An actual query response could also be helpful. In that case please include "trace.level": 4 in the query

[hmusum]

Noe feedback, closing