Add ability for minion nodes to download segments from servers during task execution

[tibrewalpratik17]

label:

• feature
• release-notes

Resolves #12458. See the linked issue for more details.

This patch allows minion nodes to download segments from servers rather than only relying on deepstore copy. This is behind a task-level config: allowDownloadFromServer (default value is false).

Tested in our cluster by removing deepstore copies explicitly and enabling this config. We were able to successfully compact a segment post this config.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 33.33333% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 62.23%. Comparing base (59551e4) to head (d67f21c). Report is 458 commits behind head on master.

Files	Patch %	Lines
...he/pinot/plugin/minion/tasks/BaseTaskExecutor.java	5.26%	18 Missing :warning:
.../tasks/BaseMultipleSegmentsConversionExecutor.java	0.00%	9 Missing :warning:
...ion/tasks/BaseSingleSegmentConversionExecutor.java	0.00%	3 Missing :warning:
...che/pinot/plugin/minion/tasks/MinionTaskUtils.java	71.42%	0 Missing and 2 partials :warning:
...psertcompaction/UpsertCompactionTaskGenerator.java	0.00%	2 Missing :warning:
.../plugin/minion/tasks/purge/PurgeTaskGenerator.java	0.00%	1 Missing :warning:
...he/pinot/segment/local/utils/TableConfigUtils.java	85.71%	0 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #12960 +/- ## ============================================ + Coverage 61.75% 62.23% +0.48% + Complexity 207 198 -9 ============================================ Files 2436 2520 +84 Lines 133233 138320 +5087 Branches 20636 21396 +760 ============================================ + Hits 82274 86081 +3807 - Misses 44911 45820 +909 - Partials 6048 6419 +371 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.20% <33.33%> (+0.49%)` | :arrow_up: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.12% <33.33%> (+0.49%)` | :arrow_up: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.21% <33.33%> (+0.47%)` | :arrow_up: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.10% <33.33%> (+34.37%)` | :arrow_up: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.23% <33.33%> (+0.48%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.22% <33.33%> (+0.48%)` | :arrow_up: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.72% <0.00%> (-0.17%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/12960/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.94% <33.33%> (+0.21%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[tibrewalpratik17]

cc @ankitsultana @Jackie-Jiang @snleee can you please help review?

[tibrewalpratik17]

Thanks @ankitsultana for the review! Addressed your comments.

[npawar]

Have the docs been updated @tibrewalpratik17 ?

[tibrewalpratik17]

Thanks @npawar for the reminder! I will update the docs with all the PRs you tagged me on.