Canner / wren-engine

🤖 The semantic engine for LLMs, bringing semantic context to AI agents. 🔥
https://getwren.ai/oss
Apache License 2.0
129 stars 34 forks source link

fix(core): disable pushdown filter to keep the casting for timestamp column #907

Closed goldmedal closed 4 days ago

goldmedal commented 5 days ago

Description

Sometimes, we will cast a timestamp column to a timestamp type because we must handle the DATETIME and TIMESTAMP for the BigQuery. However, push_down_filter will remove the casting because it can't differentiate DATETIME and TIMESTAMP in the DataFusion layer.

see the unit test for the details. If applying push_down_filter, the result will be as below. However, it will fail if 出道時間 is DATETIME in BigQuery

SELECT 
  count(*) 
FROM 
  (
    SELECT 
      artist.cast_timestamp 
    FROM 
      (
        SELECT 
          CAST(
            artist."出道時間" AS TIMESTAMP WITH TIME ZONE
          ) AS cast_timestamp 
        FROM 
          artist 
        WHERE 
          artist."出道時間" > CAST(
            '2011-01-01 21:00:00' AS TIMESTAMP
          )
      ) AS artist
  ) AS artist