[Proposal][Vega] Support PPL in vega visualization

[ruanyl]

Introduction

Currently, the OSD vega visualization only support OpenSearch query DSL. As PPL is also adopted and used by different features of OSD. I propose to also support PPL in vega visualization.

Proposal

(draft) I propose to extend the existing data.url to have a new %type%: ppl, the schema could be like:

{
  "data": {
        "url": {
            "%type%": "ppl",
            "body": {
                "query": "source=opensearch_dashboards_sample_data_logs| where QUERY_STRING(['response'], '4* OR 5*')"
            }
        }
    }
}

Open question

1. PPL is a feature from a OpenSearch plugin, but vega is a core OSD plugin feature, having OSD core to "depend"(as it needs to call PPL endpoint to get data) on a OpenSearch plugin, will this be a concern even though the OpenSearch plugin will be present in main OpenSearch distributions?

[kavilla]

The actual location of the plugin that adds this support is to be decided. But in the feature/discover-next branch this could be as easy as adding vega (or the specific app id) to the supported app names: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/feature/discover-next/plugins-extra/query_enhancements/public/plugin.tsx#L60

As long as vega utilizes the application low level search call within the search service.

to test it you just need to run OpenSearch with: yarn opensearch snapshot --sql

and within the feature/discover-next branch: yarn start --extra-plugins --data.enhancements.enabled=true

But I haven't looked into how vega is using for search

[ruanyl]

Thanks @kavilla for the information!

this could be as easy as adding vega (or the specific app id) to the supported app names

I suppose that will make the language to be available for selection in the search bar? For vega visualization, the query will be defined in vega json object, it looks like this currently, the body defines the aggs and it will be sent to /internal/search/opensearch along with the query.

{
  $schema: https://vega.github.io/schema/vega-lite/v5.json
  title: Event counts from all indexes

  // Define the data source
  data: {
    url: {
      %context%: true
      // Filter the time picker (upper right corner) with this field
      %timefield%: @timestamp
      index: _all
      body: {
        aggs: {
          time_buckets: {
            date_histogram: {
              // Use date histogram aggregation on @timestamp field
              field: @timestamp
              // The interval value will depend on the daterange picker (true), or use an integer to set an approximate bucket count
              interval: {%autointerval%: true}
              // Make sure we get an entire range, even if it has no data
              extended_bounds: {
                // Use the current time range's start and end
                min: {%timefilter%: "min"}
                max: {%timefilter%: "max"}
              }
              // Use this for linear (e.g. line, area) graphs.  Without it, empty buckets will not show up
              min_doc_count: 0
            }
          }
        }
        // Speed up the response by only including aggregation results
        size: 0
      }
    }
    format: {property: "aggregations.time_buckets.buckets"}
  }

  mark: line

  // "encoding" tells the "mark" what data to use and in what way.  See https://vega.github.io/vega-lite/docs/encoding.html
  encoding: {
    x: {
      // The "key" value is the timestamp in milliseconds.  Use it for X axis.
      field: key
      type: temporal
      axis: {title: false} // Customize X axis format
    }
    y: {
      // The "doc_count" is the count per bucket.  Use it for Y axis.
      field: doc_count
      type: quantitative
      axis: {title: "Document count"}
    }
  }
}

Similarly, I'd like to make this vega object to support ppl query.

As long as vega utilizes the application low level search call within the search service.

I'm not sure if I fully get what is low level search call, but the API to call is /api/ppl/search, is this low level search call?

[ruanyl]

Closed by https://github.com/opensearch-project/OpenSearch-Dashboards/pull/7285