optimization: add retry when clean up mark from resourcetemplate failed

[zhzhuang-zju]

What type of PR is this? /kind cleanup

What this PR does / why we need it:

Through reviewing the e2e logs, it was found that when deleting pp/cpp, there is a chance that cleaning up the labels/annotations of the resource template might fail. To address this, we can add retries to the function CleanupResourceTemplateMarks and reduce the scope and impact of these retries.

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

[codecov-commenter]

:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 16 lines in your changes missing coverage. Please review.

Project coverage is 28.25%. Comparing base (c38f169) to head (f24663b). Report is 2 commits behind head on master.

Files	Patch %	Lines
pkg/detector/detector.go	0.00%	16 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5259 +/- ## ======================================= Coverage 28.24% 28.25% ======================================= Files 632 632 Lines 43747 43747 ======================================= + Hits 12357 12361 +4 + Misses 30488 30486 -2 + Partials 902 900 -2 ``` | [Flag](https://app.codecov.io/gh/karmada-io/karmada/pull/5259/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/karmada-io/karmada/pull/5259/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | `28.25% <0.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[XiShanYongYe-Chang]

Hi @chaunceyjiang @whitewindmills can you help take a review? /cc @chaunceyjiang @whitewindmills

[whitewindmills]

/assign

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/karmada-io/karmada/blob/master/OWNERS)~~ [XiShanYongYe-Chang] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment