datafuselabs / databend

𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.31k stars 704 forks source link

feat: task_advice system table #14323

Open BohuTANG opened 4 months ago

BohuTANG commented 4 months ago

Summary

Add a system table for advice the background serivice:

catalog_name database_name table_name task_type need_run task_sql reason(variant)
default db xx COMPACT 1 optimize table xx compact limit 3 "{status in json}"
default db xx AGGREGATING_INDEX 0 refresh aggregating index xx_agg_idx limit 10 "{status in json}"
default db yy ADD_CLUSTER_KEY 1 alter table yy cluster by(col1) "{status in json}"
BohuTANG commented 4 months ago

@ZhiHanZ this task_advice table need your advices :/

flaneur2020 commented 2 weeks ago

SHOW ADVICES FOR TABLE xxx;

sundy-li commented 2 weeks ago

The compaction service needs to know the endpoints of the query nodes to fetch the result from advice table. But the query nodes are serverless and multi-tenant.

Since the compaction service must depend on S3 and metaservice, why not fetch the advice in the compaction service on schedule?

BohuTANG commented 2 weeks ago

why not fetch the advice in the compaction service on schedule?

We can execute the advices in the cloud background service. However, this issue extends beyond just background services; Aims to give some advices for the tables in databases, and tell the user how to do next.

BohuTANG commented 2 weeks ago

Reference: Oracle Database Advisors: https://blogs.oracle.com/coretec/post/oracle-database-advisors-overview