The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
In shared data mode, file gc relies on the efficiency of the delete file thread pool. The default thread pool size is too small, which causes the delete file task queue to accumulate easily under high ingestion concurrency or under large ingestion, resulting in slowing donw vacuum task.
What I'm doing:
In the pr, we slightly increase the delete file thread pool size into half the cpu core (config::drop_tablet_worker_count <= 0) If user change the config::drop_tablet_worker_count <= 0 at runtime, the thread pool size will also be modified as half the cpu core.
Fixes #issue
What type of PR is this:
[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool
Does this PR entail a change in behavior?
[x] Yes, this PR will result in a change in behavior.
[ ] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[x] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)
[ ] I have added documentation for my new feature or new function
[x] This is a backport pr
Bugfix cherry-pick branch check:
[x] I have checked the version labels which the pr will be auto-backported to the target branch
[x] 3.4
[x] 3.3
[ ] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5
This is an automatic backport of pull request #52246 done by Mergify.
Why I'm doing:
In shared data mode, file gc relies on the efficiency of the delete file thread pool. The default thread pool size is too small, which causes the delete file task queue to accumulate easily under high ingestion concurrency or under large ingestion, resulting in slowing donw vacuum task.
What I'm doing:
In the pr, we slightly increase the delete file thread pool size into half the cpu core (config::drop_tablet_worker_count <= 0) If user change the config::drop_tablet_worker_count <= 0 at runtime, the thread pool size will also be modified as half the cpu core.
Fixes #issue
What type of PR is this:
[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool
Does this PR entail a change in behavior?
[x] Yes, this PR will result in a change in behavior.
[ ] No, this PR will not result in a change in behavior.
If yes, please specify the type of change:
[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[x] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.
Checklist:
[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)
[ ] I have added documentation for my new feature or new function
Why I'm doing:
In shared data mode, file gc relies on the efficiency of the delete file thread pool. The default thread pool size is too small, which causes the delete file task queue to accumulate easily under high ingestion concurrency or under large ingestion, resulting in slowing donw vacuum task.
What I'm doing:
In the pr, we slightly increase the delete file thread pool size into half the cpu core (
config::drop_tablet_worker_count <= 0) If user change theconfig::drop_tablet_worker_count <= 0at runtime, the thread pool size will also be modified as half the cpu core.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
This is an automatic backport of pull request #52246 done by Mergify.
Why I'm doing:
In shared data mode, file gc relies on the efficiency of the delete file thread pool. The default thread pool size is too small, which causes the delete file task queue to accumulate easily under high ingestion concurrency or under large ingestion, resulting in slowing donw vacuum task.
What I'm doing:
In the pr, we slightly increase the delete file thread pool size into half the cpu core (
config::drop_tablet_worker_count <= 0) If user change theconfig::drop_tablet_worker_count <= 0at runtime, the thread pool size will also be modified as half the cpu core.Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist: