opensearch-project / sql

Query your data using familiar SQL or intuitive Piped Processing Language (PPL)
https://opensearch.org/docs/latest/search-plugins/sql/index/
Apache License 2.0
116 stars 134 forks source link

[BUG] Enhance OS value parsing #3001

Open bugmakerrrrrr opened 2 weeks ago

bugmakerrrrrr commented 2 weeks ago

What is the bug? In OS, the numeric/boolean fields can be indexed in text format, and the string field can be indexed in numeric format, but the sql plugin cannot parse these value correctly.

How can one reproduce the bug?

PUT test
{
  "mappings": {
    "properties": {
      "field1": {
        "type": "long"
      },
      "field2": {
        "type": "boolean"
      }
    }
  }
}

POST test/_bulk?refresh=true
{"index": {}}
{"field1": ["1", 1], "field2": ["true", true]}
{"index": {}}
{"field1": "1", "field2": "true"}

POST _plugins/_sql
{
  "query": "select field1, field2 from test"
}

#response
{
  "schema": [
    {
      "name": "field1",
      "type": "long"
    },
    {
      "name": "field2",
      "type": "boolean"
    }
  ],
  "datarows": [
    [
      "1",
      "true"
    ],
    [
      0,
      false
    ]
  ],
  "total": 2,
  "size": 2,
  "status": 200
}
dai-chen commented 2 weeks ago

PPL (or SQL v2) shows numerical field with string value as 0.

POST num_with_string_value/_doc
{
  "number": 123
}

POST num_with_string_value/_doc
{
  "number": "456"
}

POST _plugins/_ppl
{
  "query": "source = num_with_string_value | where number > 0"
}
{
  "schema": [
    {
      "name": "number",
      "type": "long"
    }
  ],
  "datarows": [
    [
      0
    ],
    [
      123
    ]
  ],
  "total": 2,
  "size": 2
}