search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.17k stars 1.21k forks source link

Bugfix: Validate minionInstanceTag during task-generation #13092

Closed tibrewalpratik17 closed 1 week ago

tibrewalpratik17 commented 1 week ago

label: bugfix minion

This patch validates whether the configured minionInstanceTag actually exists in the list of instance-tags or not. Before this, if a wrong instanceTag was configured for a task, Pinot scheduled those tasks and they got stuck in IN_PROGRESS state. Until the task is timed out, no another task scheduling for that table and task type was possible. Or you need to update some minion instance with the specified instanceTag. Ideally, we should error out by throwing an exception for instanceTag not found which this patch achieves.

There is a more proactive way of doing this at table-config updation itself. We will try to achieve that when we introduce minionTenant config itself.

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 62.20%. Comparing base (59551e4) to head (a0047f1). Report is 419 commits behind head on master.

Files Patch % Lines
...controller/helix/core/minion/PinotTaskManager.java 0.00% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #13092 +/- ## ============================================ + Coverage 61.75% 62.20% +0.45% + Complexity 207 198 -9 ============================================ Files 2436 2514 +78 Lines 133233 137793 +4560 Branches 20636 21321 +685 ============================================ + Hits 82274 85715 +3441 - Misses 44911 45683 +772 - Partials 6048 6395 +347 ``` | [Flag](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration1](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: | | [integration2](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | | | [java-11](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.15% <0.00%> (+0.44%)` | :arrow_up: | | [java-21](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.07% <0.00%> (+0.44%)` | :arrow_up: | | [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.19% <0.00%> (+0.44%)` | :arrow_up: | | [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.03% <0.00%> (+34.30%)` | :arrow_up: | | [temurin](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.20% <0.00%> (+0.45%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.20% <0.00%> (+0.45%)` | :arrow_up: | | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `46.86% <ø> (-0.03%)` | :arrow_down: | | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13092/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.76% <0.00%> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ankitsultana commented 1 week ago

@tibrewalpratik17 : let's also emit numErrorTasksScheduled metric in a follow-up PR so we can setup alerting/monitoring