datalens-tech / datalens

A modern, scalable analytics system
https://datalens.tech
Apache License 2.0
1.37k stars 60 forks source link

Difficult to work with large tables #12

Open alexey-milovidov opened 1 year ago

alexey-milovidov commented 1 year ago

I defined a dataset on a table with 21 billion records. When I tried to create my first chart, it did this:

![Screenshot_20230926_190229](https://github.com/datalens-tech/datalens/assets/18581488/d81e4044-9b75-4eff-8107-98b6f15dd4ce)

Running this query on ClickHouse:

SELECT `t1`.`created_at` AS `res_0` FROM `default`.`twitter` AS `t1` GROUP BY `res_0` LIMIT 1000001

The query will process nicely, but the aggregation is performed by every unique time point, and it requires 94.5 seconds to finish on a service in ClickHouse Cloud.

resure commented 1 year ago

Grouping can be changed in the field settings (click on the icon inside the field -> Grouping

Screenshot 2023-09-29 at 16 26 57

But current default grouping is certainly not useful, especially when no filters are specified, we'll probably switch it to day or week.