support auto delete WorkloadRebalancer when time up

chaosi-zju commented 2 weeks ago

What type of PR is this?

/kind feature

What this PR does / why we need it:

Support auto delete WorkloadRebalancer when time up:

referring to Automatic Cleanup for Finished Jobs.

Introduces field ttlSecondsAfterFinished which limits the lifetime of a WorkloadRebalancer that has finished execution (finished execution means each target workload is finished with result of Successful or Failed).

If this field is set, ttlSecondsAfterFinished after the WorkloadRebalancer finishes, it is eligible to be automatically deleted.
If this field is unset, the WorkloadRebalancer won't be automatically deleted.
If this field is set to zero, the WorkloadRebalancer becomes eligible to be deleted immediately after it finishes.

Considering several corner cases:

case 1: if a new target workload was added into WorkloadRebalancer before ttlSecondsAfterFinished expired, which means the finish time of the WorkloadRebalancer is refreshed, so the delete action is deferred since expire time is refreshed too.
case 2: if ttlSecondsAfterFinished is modified before ttlSecondsAfterFinished expired, delete action should be performed according to latest ttlSecondsAfterFinished.
case 3: when we have got and checked latest WorkloadRebalancer object and try to delete it, if a modification to WorkloadRebalancer occurred just right between the two time point, the previous delete action should be Interrupted.

Several key implementation:

A WorkloadRebalancer is judged as finished should meet two requirements:
- all expected workloads are finished with result of Successful or Failed.
- introduce a new field named ObservedGeneration to Status of WorkloadRebalancer, and it should be equal to .metadata.Generation, to prevent that the WorkloadRebalancer is updated but controller hasn't in time refreshed its Status.
When WorkloadRebalancer is Created or Updated, add it to the workqueue and calculate its expiring time, and call workqueue.AddAfter() function to re-enqueue it once more if it hasn't expired.
Before deleting the WorkloadRebalancer, do a final sanity check. Use the latest WorkloadRebalancer directly fetched from api server to see if the TTL truly expires, rather than object from lister cache.
When deleting the WorkloadRebalancer, it is needed to confirm that the resourceVersion of the deleted object is as expected, to prevent from above corner case 3.

Which issue(s) this PR fixes:

Fixes part of #4840

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

support auto delete WorkloadRebalancer when time up

chaosi-zju commented 2 weeks ago

/hold

for #4875 #4860

chaosi-zju commented 1 week ago

/hold cancel

codecov-commenter commented 1 week ago

Codecov Report

Attention: Patch coverage is 9.21053% with 69 lines in your changes are missing coverage. Please review.

Project coverage is 53.07%. Comparing base (6cfed59) to head (98845e0). Report is 8 commits behind head on master.

Files	Patch %	Lines
...lers/workloadrebalancer/ttlafterfinished_worker.go	0.00%	52 Missing :warning:
...orkloadrebalancer/workloadrebalancer_controller.go	29.16%	13 Missing and 4 partials :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #4894 +/- ## ========================================== - Coverage 53.19% 53.07% -0.12% ========================================== Files 252 253 +1 Lines 20509 20586 +77 ========================================== + Hits 10909 10926 +17 - Misses 8880 8933 +53 - Partials 720 727 +7 ``` | [Flag](https://app.codecov.io/gh/karmada-io/karmada/pull/4894/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/karmada-io/karmada/pull/4894/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | `53.07% <9.21%> (-0.12%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

chaosi-zju commented 5 days ago

CC @RainbowMango can this begin to review?

RainbowMango commented 4 days ago

/assign

By the way.

If this field is set, ttlSecondsAfterFinished after the WorkloadRebalancer finishes, it is eligible to be automatically deleted.

ttlSecondsAfterFinished --> ttlMinutesAfterFinished in the PR description.

karmada-bot commented 3 days ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: To complete the pull request process, please ask for approval from rainbowmango after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/karmada-io/karmada/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

karmada-io / karmada

support auto delete WorkloadRebalancer when time up #4894

Codecov Report