Intermittent 500 error during search: Failed querying '[]': Index 124 out of bounds for length 100

[nehajatav]

Describe the bug

{
  "yql" : "select * from sources * where ((collection_attr matches \"^some\\\\.metadata\\\\..*\" OR collection contains \"some.metadata\" OR collection_attr matches \"^some\\\\.content\\\\..*\" OR collection contains \"some.content\" OR collection_attr matches \"^some\\\\.blah\\\\_scratch\\\\_pad\\\\..*\" OR collection contains \"some.blah_scratch_pad\" OR collection_attr matches \"^some\\\\.foo\\\\..*\" OR collection contains \"some.foo\" OR collection_attr matches \"^some\\\\.certifications\\\\..*\" OR collection contains \"some.certifications\" OR collection contains \"public\") AND content_blah contains \"Test-unique-doc-879\") order by '[relevance]' DESC,'[docid]' DESC limit 0 offset 0",
  "traceLevel" : 2,
  "ranking.profile" : "some-rank-profile",
  "presentation" : {
    "summary" : "some-summary"
  }
}

[b5aa945f381e56ad/9355e40cf1ea9a84] [ERROR] 2024-01-01 14:47:16.284 JSON response -> getRoot().getErrors().getMessage() evaluated to "Failed querying '[]': Index 124 out of bounds for length 100"

JsonAccessLog.default.20240101194122:{"ip":"","time":1704138436.228,"duration":0.052,"responsesize":62646,"requestsize":1574,"code":500,"method":"POST","uri":"/search/","version":"HTTP/2.0","agent":"b5aa945f381e56ad","host":"vespa-config.example.com","scheme":"https","localport":8443,"ssl-principal":"CN=vespa-config.example.com,OU=SOME OU,O=ABC,L=XXYZ,ST=XYZ,C=MM","peeraddr":"","peerport":41858,"search":{"totalhits":1,"hits":1,"coverage":{"coverage":100,"documents":1563757}},"attributes":{"http2-stream-id":"1117"}}

To Reproduce Unable to reproduce

Expected behavior Atleast expect some clue to what is the issue

Environment (please complete the following information):

• OS: Red Hat Enterprise Linux release 8.6 (Ootpa)
• Infrastructure: Podman 4.2.0

Vespa version vespaengine/vespa:8.221.29

Additional context No stderr or error in container, content or config logs

[bratseth]

What in the query is there 124 of? Do you have the stacktrace? If not: Do you have custom searchers?

[nehajatav]

We dont use custom searchers. Does vespa search return stacktrace in response string? I suppose above is coming from below excerpt of com.yahoo.prelude.cluster.ClusterSearcher mergedResult.hits().addError(ErrorMessage.createInternalServerError("Failed querying '" + query.getModel().getRestrict() + "': " + Exceptions.toMessageString(e)));

[bratseth]

You're right and we should have kept the track trace in this case. I've done that in https://github.com/vespa-engine/vespa/pull/29782, should be available by Friday.

The other way to figure out the cause is to check what you have 124 of in this query.