enhance: Increase task capacity and clean illegal task

[XuanYang-cn]

1. taskQueueCapacity 256 is too small for production when we want to re-write the entire collection

2. tasks should be cleaned when unable to recover, or the meta will remain in etcd forever later.

[czs007]

/approve /lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[configs/OWNERS](https://github.com/milvus-io/milvus/blob/master/configs/OWNERS)~~ [czs007] - ~~[internal/datacoord/OWNERS](https://github.com/milvus-io/milvus/blob/master/internal/datacoord/OWNERS)~~ [czs007] - ~~[pkg/util/OWNERS](https://github.com/milvus-io/milvus/blob/master/pkg/util/OWNERS)~~ [czs007] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 78.99%. Comparing base (b983ef9) to head (0b0179f). Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/datacoord/compaction.go	0.00%	3 Missing :warning:

Additional details and impacted files

[](https://app.codecov.io/gh/milvus-io/milvus/pull/37896?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) ```diff @@ Coverage Diff @@ ## master #37896 +/- ## ========================================== - Coverage 81.03% 78.99% -2.05% ========================================== Files 1357 1349 -8 Lines 190182 188905 -1277 ========================================== - Hits 154121 149232 -4889 - Misses 30591 34238 +3647 + Partials 5470 5435 -35 ``` | [Components](https://app.codecov.io/gh/milvus-io/milvus/pull/37896/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | Coverage Δ | | |---|---|---| | [Client](https://app.codecov.io/gh/milvus-io/milvus/pull/37896/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | `72.16% <ø> (ø)` | | | [Core](https://app.codecov.io/gh/milvus-io/milvus/pull/37896/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | `68.87% <ø> (ø)` | | | [Go](https://app.codecov.io/gh/milvus-io/milvus/pull/37896/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | `80.78% <25.00%> (-2.43%)` | :arrow_down: | | [Files with missing lines](https://app.codecov.io/gh/milvus-io/milvus/pull/37896?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io) | Coverage Δ | | |---|---|---| | [pkg/util/paramtable/component\_param.go](https://app.codecov.io/gh/milvus-io/milvus/pull/37896?src=pr&el=tree&filepath=pkg%2Futil%2Fparamtable%2Fcomponent_param.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-cGtnL3V0aWwvcGFyYW10YWJsZS9jb21wb25lbnRfcGFyYW0uZ28=) | `98.35% <100.00%> (ø)` | | | [internal/datacoord/compaction.go](https://app.codecov.io/gh/milvus-io/milvus/pull/37896?src=pr&el=tree&filepath=internal%2Fdatacoord%2Fcompaction.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io#diff-aW50ZXJuYWwvZGF0YWNvb3JkL2NvbXBhY3Rpb24uZ28=) | `66.09% <0.00%> (-5.70%)` | :arrow_down: | ... and [254 files with indirect coverage changes](https://app.codecov.io/gh/milvus-io/milvus/pull/37896/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=milvus-io)

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests