Databend Task Frameworks

[Xuanwo]

Summary

Databend needs task frameworks so that it can:

• Schedule cron jobs like compation, reculster
• Create/Cancel/Start/Stop long-time job like copy/reculster
• Create/Alter/Delete user's job like copy/delete

[ZhiHanZ]

I think besides those features, another feature value most is the ability to handle tasks in short-lived warehouse running instances, and we may need:

1. sync and manage task session to get avoid of session leakage. (/query/status response should have dedicates struct represents started/running/finished jobs and tasks)
2. ability to drain task in a single instance and distribute the remaining tasks into the rest of active instances (fault tolerance)

cc @flaneur2020 any other suggestions?

[b41sh]

@Xuanwo Is there any progress on this issue? It is time-consuming to build BlockFilter and BlockStatistics when inserting a large number of values. If we can use an asynchronous task framework to do this can speed up the insertion operation.

[Xuanwo]

Is there any progress on this issue?

No progress so far.