Add `_source` option for elasticsearch plugin to retrieve only defined fields when using Metric: Raw data

[kareig]

Hi,

What would you like to be added:

Would be very nice to have a possibility to define _source for elasticsearch plugin when Metric: Raw Data & Visualization Table is being used. Currently _source can not be defined within Lucene query itself because it is outside of it.

Why is this needed:

For presenting, retrieving from elasticsearch only these _doc fields which are defined in Grafana and not all raw data _doc fields.

Testing:

Current Grafana query:

request:Object
url:"api/datasources/proxy/28/_msearch"
method:"POST"
data:"{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"","max_concurrent_shard_requests":256}
{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1621491369329,"lte":1621577769330,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"_index:myindex-* AND State:\"OPEN\""}}]}},"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}
"

Response from ELK:

         {
            "_index" : "myindex-2021-05-20-000001",
            "_type" : "fluentd",
            "_id" : "jozBiXkBtz4lpUEu1ZQp",
            "_score" : null,
            "_source" : {
              "State" : "OPEN",
              "level" : "info",
              "msg" : "mymessage",
              "time" : "2021-05-20T14:30:45+02:00",
              "@timestamp" : "2021-05-20T14:30:45.000000000+02:00"
            },

Modified Grafana query tested on elasticsearch by adding "_source":"State":

POST _msearch
{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"","max_concurrent_shard_requests":256}
{"size":500,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":1621491369329,"lte":1621577769330,"format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":"_index:myindex-* AND State:\"OPEN\""}}]}},"_source":"State","sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}},{"_doc":{"order":"desc"}}],"script_fields":{}}

Response from ELK:

          {
            "_index" : "myindex-2021-05-20-000001",
            "_type" : "fluentd",
            "_id" : "jozBiXkBtz4lpUEu1ZQp",
            "_score" : null,
            "_source" : {
              "State" : "OPEN"
            },

Thanks

[matyax]

We need to follow up on this one.