search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
9.21k stars 1.83k forks source link

Support temporary table in TableQueryPlanAction #49356

Open dohongdayi opened 3 months ago

dohongdayi commented 3 months ago

Feature request

Is your feature request related to a problem? Please describe.

We are using starrocks-spark-connector to get computation result from starrocks, and we have a lot of spark ETL jobs but not enough disk space for that amount of materialized views, in which case we have to create and drop starrocks regular table each time running each spark ETL job, since TableQueryPlanAction only supports regular olap tables.

Describe the solution you'd like

If TableQueryPlanAction also supported temporary tables, which could be read by starrocks-spark-connector and be automatically dropped by starrocks after spark job completed, it could improve user experience and disk utilization.

Describe alternatives you've considered

We have considered materialized views, but we have a lot of spark jobs but not enough disk space.

Additional context

Starrocks temporary table is also olap table, that's exactly TableQueryPlanAction only supports, I don't see any reason why we not support them as well.

jaogoy commented 3 months ago

@dohongdayi What's TableQueryPlanAction.