Configurable query logging using search pipelines

[msfroh]

Is your feature request related to a problem? Please describe. On https://github.com/opensearch-project/OpenSearch/pull/10650, I tried to derail things a little by asking "Couldn't these coordinator-level slow logs just be implemented as a SearchResponseProcessor? That would keep the code out of OpenSearch core (at least out of /server), be much more configurable (since search pipelines JSON config is way better than flat index settings), and could integrate nicely with cool future ideas like views.

That said, I could understand the argument for making coordinator slow logs "look like" shard-level slow logs, though I respectfully think they're both wrong. Consistently wrong is still consistent, and consistency is a good thing.

Describe the solution you'd like Let's build a new, better query logging component using search pipelines!

It will be better than the existing slow logs, because it won't be limited to just slow logs. You can log any query you want, with orthogonal selector logic. Maybe you want to log any queries with "expensive" clauses (e.g. wildcards). Maybe you want to log all queries that had zero matches. You can log any query that contains the word "moose" (if you're so inclined).

Additionally, I don't want this logging component to have to output to a file. It should support configurable sinks. For v1, I would ship with a sink that writes to an OpenSearch index. You can use OpenSearch to search your query logs, since I hear that OpenSearch is pretty good for searching logs.

Describe alternatives you've considered Arguably, this is the alternative to something like coordinator-level slow logs (e.g. https://github.com/opensearch-project/OpenSearch/pull/10650).

Additional context N/A

[msfroh]

I'm imagining a simple slow-logging configuration could be something like:

PUT /_search/pipeline/logging_pipeline
{
  "response_processors" : [
    {
      "query_log" : {
        "if" : "_ctx.rsp.took > 100",
        "target" : {
          "index" : "slow_logs"
        }
      }
    }
  ]
}

This assumes a Painless script reading the took property from the SearchResponse. Alternatively, we could probably build some conditional primitives that the processor would handle directly.

I could imagine the following response properties being useful in a logging predicate:

1. Took time (overall)
2. Total hits
3. Number of returned hits
4. Probably per-phase took time (see https://github.com/opensearch-project/OpenSearch/issues/7334).

Predicates that match against the search request would be nice, but require a bit more work. I would like something that uses the QueryBuilderVisitor to traverse a complex query and identify specific query clauses. I discussed some ideas around restricting query clauses in https://github.com/opensearch-project/OpenSearch/issues/10938. We could reuse that kind of logic to match queries for logging.