search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
6.88k stars 569 forks source link

feat(compaction): support batch get compact task #11790

Open Li0k opened 1 year ago

Li0k commented 1 year ago

Is your feature request related to a problem? Please describe.

The computing resource consumption of Picker is proportional to the number of files. Therefore, when the amount of files is large, it will cause a large waste of resources to perform a complete calculation and only obtain one task.

Describe the solution you'd like

Based on the Pull Based model, it is possible to specify the number of compact tasks required at a time. Therefore it is possible to batch fetch tasks directly in a round of computation to reduce the consumption of resources.

Describe alternatives you've considered

No response

Additional context

No response

hzxa21 commented 12 months ago

@Li0k Is this issue still planned?

Li0k commented 11 months ago

@Li0k Is this issue still planned?

move it to next release

hzxa21 commented 11 months ago

@hzxa21 Any updates?

Li0k commented 11 months ago

@hzxa21 Any updates?

no update , suspend it.

github-actions[bot] commented 3 months ago

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean. Don't worry if you think the issue is still valuable to continue in the future. It's searchable and can be reopened when it's time. 😄

Li0k commented 3 months ago

The batch get task across compaction groups has been implemented, further implementation of the batch get task within compaction groups will require code refactoring, and I would like to keep this issue.