search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
9.03k stars 1.82k forks source link

Support cooldown to external lake table #51649

Open hoffermei opened 1 month ago

hoffermei commented 1 month ago

Feature request

Is your feature request related to a problem? Please describe.

As data stored in olap table and the limit of local storage, we offten set a ttl(dynamic.start property) for local partition. When a partition reach it's ttl time, it will be delete by dynamic partition scheduler. If we config a table with big ttl, we must use more machine to store history data.

Another case is we sometimes need do offline analysis, expect to be independent with real time compute.

Describe the solution you'd like

Add a mechanism to cooldown olap table partition to external lake table, then history data could be save to lake table, offline analysis can be do on lake table with independent compute nodes.

Describe alternatives you've considered

Additional context

hoffermei commented 1 month ago

pr #45536

LiShuMing commented 1 month ago

After cooldown, what's the behavior of querying partitions (cooldown) ?