opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
22 stars 33 forks source link

[BUG] timestamp comparison using "INTERVAL" literal doesn't work against ingested data #763

Open A-Gray-Cat opened 1 month ago

A-Gray-Cat commented 1 month ago

What is the bug? When running a direct query, the following query would work and return results

SELECT *
FROM table1
WHERE time_dt BETWEEN CURRENT_TIMESTAMP - INTERVAL '7' DAY AND CURRENT_TIMESTAMP
LIMIT 10

However, when running the same query against a materialized view, covering index, or ingested data, it would error out. After removing the timestamp where clause, the query would go through: Bad Request, this query is not runnable.

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Add WHERE timestamp BETWEEN CURRENT_TIMESTAMP - INTERVAL '7' DAY AND CURRENT_TIMESTAMP to any query, and run it against a flint table.
  2. Create a materialized view from the same flint table.
  3. Run the same query against the materialized view you just created, and the timestamp filter will make the query error out.

What is the expected behavior? The timestamp comparison should work for both flint and local index.

What is your host/environment?

Do you have any screenshots? If applicable, add screenshots to help explain your problem.

Do you have any additional context? Add any other context about the problem.

YANG-DB commented 1 month ago

@penghuo @dai-chen could u plz take a look ?