insights: better surface errors

[slimsag]

The backend's queryrunner runs search queries here, and there are a number of reasons that a search query could result in an error where "retrying later" is not strictly an obvious option and we have to decide what to do:

• Repositories are being cloned, are "missing", or timed out -> today we log a warning to repo-updater, and record whatever results we got. No retry will happen ever again. (fixing this could possibly help resolve this.)
• We hit a search limit when searching. Theoretically should never happen because we specify count:9999999 in all queries - but could happen if search is really strained for resources I guess. In this case, we log an error, record what we got, and never retry.
• The user actually messed up their query. This is perhaps the most concerning, and would only be surfaced in repo-updater logs which nobody looks at. Need a way to surface this info to the user in the web UI.
  • https://sourcegraph.com/github.com/sourcegraph/sourcegraph@61e8253febdee226c940a81c467affb0d4d0b809/-/blob/enterprise/internal/insights/background/queryrunner/work_handler.go#L52-55
  • https://sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/enterprise/internal/insights/background/queryrunner/work_handler.go#L69-72

the function where these errors are produced and handled: https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+lang:go+file:insights/background/queryrunner+content:%22%29+Handle%28%22&patternType=literal

[github-actions[bot]]

Heads up @joelkw @felixfbecker - the "team/code-insights" label was applied to this issue.

[coury-clark]

We hit a search limit when searching. Theoretically should never happen because we specify count:9999999 in all queries - but could happen if search is really strained for resources I guess. In this case, we log an error, record what we got, and never retry.

Just a note, any queries that hit a limit are now flagged as dirty and exposed as metadata through the insights GraphQL API.

Leaving this issue open to address points 1, 3.