search

StarRocks / starrocks

StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries.
https://starrocks.io
Apache License 2.0
8.75k stars 1.75k forks source link

Support temporary table in TableQueryPlanAction #49356

Open dohongdayi opened 2 months ago

dohongdayi commented 2 months ago

Feature request

Is your feature request related to a problem? Please describe.

We are using starrocks-spark-connector to get computation result from starrocks, and we have a lot of spark ETL jobs but not enough disk space for that amount of materialized views, in which case we have to create and drop starrocks regular table each time running each spark ETL job, since TableQueryPlanAction only supports regular olap tables.

Describe the solution you'd like

If TableQueryPlanAction also supported temporary tables, which could be read by starrocks-spark-connector and be automatically dropped by starrocks after spark job completed, it could improve user experience and disk utilization.

Describe alternatives you've considered

We have considered materialized views, but we have a lot of spark jobs but not enough disk space.

Additional context

Starrocks temporary table is also olap table, that's exactly TableQueryPlanAction only supports, I don't see any reason why we not support them as well.

jaogoy commented 2 months ago

@dohongdayi What's TableQueryPlanAction.