QuesmaOrg / quesma

Programmable database gateway
https://quesma.com
Other
108 stars 6 forks source link

Invalid total count - array type related #972

Open nablaone opened 1 week ago

nablaone commented 1 week ago

When we call Quesma

POST localhost:8080/kibana_sample_data_ecommerce/_search
Content type: application/json

{
  "_source": {
    "excludes": []
  },
  "aggs": {
    "0": {
      "terms": {
        "field": "products.product_name.keyword",
        "order": {
          "_count": "desc"
        },
        "shard_size": 25,
        "size": 5
      }
    }
  },
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "customer_birth_date",
      "format": "date_time"
    },
    {
      "field": "order_date",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "filter": [

      ],
      "must": [],
      "must_not": [],
      "should": []
    }
  },
  "runtime_mappings": {},
  "script_fields": {},
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "track_total_hits": true
}

The total hits result is incorrect. Quesma returns 10087 while elasticsearch: 4675.

Quesma executes the following SQL query:

SELECT 
 sum(count(*)) OVER () AS "metric____quesma_total_count_col_0",
 sum(count(*)) OVER () AS "aggr__0__parent_count",
 arrayJoin("products_product_name") AS "aggr__0__key_0",
 count(*) AS "aggr__0__count"
FROM default.kibana_sample_data_ecommerce
GROUP BY arrayJoin("products_product_name") AS "aggr__0__key_0"
ORDER BY "aggr__0__count" DESC, "aggr__0__key_0" ASC
LIMIT 6

Aggregation is computed OK, but computing the total count should compensate for the arrayJoin rows explosion.