opensearch-project / sql

Query your data using familiar SQL or intuitive Piped Processing Language (PPL)
https://opensearch.org/docs/latest/search-plugins/sql/index/
Apache License 2.0
119 stars 138 forks source link

[FEATURE] Support push down Sort/Filter into TableScanBuilder through Project #2894

Open qianheng-aws opened 3 months ago

qianheng-aws commented 3 months ago

Is your feature request related to a problem? Sort or Filter cannot be pushed down into TableScan due to Project.

POST _plugins/_ppl/_explain
{
  "query": """
    source=opensearch_dashboards_sample_data_flights | fields FlightTimeMin |where FlightTimeMin > 0  |  sort FlightTimeMin 
  """
}

# response

{
  "root": {
    "name": "ProjectOperator",
    "description": {
      "fields": "[FlightTimeMin]"
    },
    "children": [
      {
        "name": "SortOperator",
        "description": {
          "sortList": {
            "FlightTimeMin": {
              "sortOrder": "ASC",
              "nullOrder": "NULL_FIRST"
            }
          }
        },
        "children": [
          {
            "name": "FilterOperator",
            "description": {
              "conditions": ">(FlightTimeMin, 0)"
            },
            "children": [
              {
                "name": "ProjectOperator",
                "description": {
                  "fields": "[FlightTimeMin]"
                },
                "children": [
                  {
                    "name": "OpenSearchIndexScan",
                    "description": {
                      "request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":200,"timeout":"1m","_source":{"includes":["FlightTimeMin"],"excludes":[]}}, searchDone=false)"""
                    },
                    "children": []
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

What solution would you like? Option1: remove ProjectOperator after pushed it down into TableScan. Not clear of the background why we intentionally wants to keep it here. https://github.com/opensearch-project/sql/blob/14a80a95fb5fa36781b46f28ba52d406927e21c0/opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/OpenSearchIndexScanQueryBuilder.java#L79-L83

Option2: support push down Sort/Filter under Project.

What alternatives have you considered?

Do you have any additional context? Add any other context or screenshots about the feature request here.

dblock commented 2 months ago

[Catch All Triage - 1, 2, 3, 4, 5]