add metrics for recreate/update resource event when sync work status

[chaosi-zju]

What type of PR is this?

/kind feature

What this PR does / why we need it:

add metrics for recreate/update resource event when sync work status

detail backgroud:

When I create a sa and a sa type secret in control plane, yet, I only propagate the secret to member clusters, then the secret will be deleted immediately by sa-controller of the member cluster. In turn the work_status_controller will watch the deletion event and recreate the resource to cluster, this cycle goes on and never stops.

Normally, there shouldn't be a large number of recreate events, so, I want to record a metric of the recreate event which can monitor the abnormal situation.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Test report:

# HELP recreate_resource_to_cluster Number of recreating operation of the resource to a target member cluster. By the result, 'error' means a resource recreated failed. Otherwise 'success'. Cluster means the target member cluster.
# TYPE recreate_resource_to_cluster counter
recreate_resource_to_cluster{apigroup="v1",cluster="member1",kind="Secret",result="success"} 810
recreate_resource_to_cluster{apigroup="v1",cluster="member2",kind="Secret",result="success"} 809
# HELP update_resource_to_cluster Number of updating operation of the resource to a target member cluster. By the result, 'error' means a resource updated failed. Otherwise 'success'. Cluster means the target member cluster.
# TYPE update_resource_to_cluster counter
update_resource_to_cluster{apigroup="rbac.authorization.k8s.io/v1",cluster="member1",kind="ClusterRole",result="success"} 2
update_resource_to_cluster{apigroup="rbac.authorization.k8s.io/v1",cluster="member2",kind="ClusterRole",result="success"} 1

Does this PR introduce a user-facing change?:

karmada-controller-manager: add metrics for recreate/update resource event when sync work status

[codecov-commenter]

:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 28.26%. Comparing base (bc1c96e) to head (32850c4). Report is 10 commits behind head on master.

Files	Patch %	Lines
pkg/controllers/status/work_status_controller.go	0.00%	6 Missing :warning:
pkg/metrics/resource.go	0.00%	6 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #5247 +/- ## ========================================== - Coverage 28.26% 28.26% -0.01% ========================================== Files 632 632 Lines 43732 43741 +9 ========================================== + Hits 12360 12362 +2 - Misses 30470 30479 +9 + Partials 902 900 -2 ``` | [Flag](https://app.codecov.io/gh/karmada-io/karmada/pull/5247/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/karmada-io/karmada/pull/5247/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io) | `28.26% <0.00%> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=karmada-io#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[XiShanYongYe-Chang]

Hi @chaunceyjiang @whitewindmills would you like to have a look? /cc @chaunceyjiang @whitewindmills

[XiShanYongYe-Chang]

How about updating the release note like this:

karmada-controller-manager/karmada-agent: add metrics for recreate/update resource event when sync work status

[XiShanYongYe-Chang]

Thanks~ /lgtm

[chaosi-zju]

/retest

[chaosi-zju]

hi @whitewindmills, do you have any further comments?

[whitewindmills]

/lgtm

[XiShanYongYe-Chang]

/approve

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: XiShanYongYe-Chang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/karmada-io/karmada/blob/master/OWNERS)~~ [XiShanYongYe-Chang] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[XiShanYongYe-Chang]

Can you help add those new metrics in the website?