opensearch-project / sql

Query your data using familiar SQL or intuitive Piped Processing Language (PPL)
https://opensearch.org/docs/latest/search-plugins/sql/index/
Apache License 2.0
121 stars 140 forks source link

Fix: Date field format parsing for legacy query engine #3160

Open andy-k-improving opened 2 days ago

andy-k-improving commented 2 days ago

Description

This is a MR, which aim to fix the issue described over https://github.com/opensearch-project/sql/issues/1545 to align the representation of date field between the legacy engine and the v2 engine.

Code changes highlight:

Although issue is being reported https://github.com/opensearch-project/sql/issues/1545 under nested-query, however this bug has wider impact outside of nested-query, and the conditions to trigger are:

When this happen, the current behaviour will only parse the first date field from the resultset.

Reference doc:

Related Issues

Resolves https://github.com/opensearch-project/sql/issues/1545

Testing plan

# Create table
PUT http://localhost:9200/datetype_test?pretty

{
  "mappings": {
    "properties": {
      "dateAsDate": {"type": "date"},
      "longAsDate": {"type": "date"},      
      "id": { "type": "long" },
      "projects": {
        "type": "nested", 
        "properties": {
          "name": { "type": "text", "fields": {"keyword": {"type": "keyword"   }
            }
          }}}}}}

# Insert data
POST http://localhost:9200/datetype_test/_bulk?refresh

{"index":{"_id":"1"}}
{"id":3,"dateAsDate":"2023-01-01T00:00:00", "longAsDate": 1672531200000, "projects":[{"name":"SQL Spectrum querying"},{"name":"SQL security"},{"name":"OpenSearch security"}]}
{"index":{"_id":"2"}}
{"id":4,"dateAsDate":"2024-01-01T00:00:00", "longAsDate": 1704067200000, "projects":[]}
{"index":{"_id":"3"}}
{"id":6,"dateAsDate":"2025-01-01T00:00:00","longAsDate": 1735689600000, "projects":[{"name":"SQL security"},{"name":"Hello security"}]}

#Qeury, both fields should be in simple date format.
POST http://localhost:9200/_plugins/_sql

{
  "query" : "SELECT t.id, t.dateAsDate, t.longAsDate FROM datetype_test AS t, t.projects AS p"
}

Check List

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

andy-k-improving commented 1 day ago

@parked-toes, @dai-chen and @anasalkouz Would you mind to have a look on this? Thanks!