search

StarRocks / starrocks

StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries.
https://starrocks.io
Apache License 2.0
8.46k stars 1.71k forks source link

spilling support in CTE #44120

Open mingmxu opened 3 months ago

mingmxu commented 3 months ago

Feature request

Is your feature request related to a problem? Please describe. Without spilling support in CTE , it becomes a double-edged sword. When it’s turned on, it might use lots of memory; however when it’s turned off, some queries might fail without the temporal result cache.

Describe the solution you'd like Support spilling_to_disk in CTE to limit the memory usage, by offloading/reloading extra data to disk;

Describe alternatives you've considered

Additional context

kangkaisen commented 3 months ago

We will support it

silverbullet233 commented 3 months ago

Could you give us more details about your usage scenario? for example, in your scenario, how big is the intermediate result produced by CTE? @mingmxu

mingmxu commented 3 months ago

Could you give us more details about your usage scenario? for example, in your scenario, how big is the intermediate result produced by CTE? @mingmxu

Can you ping me on slack? I can generate a query profile for details

stdpain commented 3 weeks ago

supported in https://github.com/StarRocks/starrocks/pull/47982