Feat/enable snapshot counts for observation

[alswl]

Problems:

When I initialize argo controller in huge cluster(more then 10000 nodes), it will takes about 1 hour to get ready.

I can only observe the progress by GetClusterInfo, but this function is blocking by a lock, so I cannot get the verbose progress.

By using this GetClusterInfoInstant, I can found how many resources is loaded by argocd controller(though this is not exactly).

Them metrics name is(with argocd metrics modified):

# HELP argocd_cluster_cache_resources_count Number of cluster cache resources
# TYPE argocd_cluster_cache_resources_count gauge
argocd_cluster_cache_resources_count{cluster="https://foo.bar:6443"} 359

# HELP argocd_cluster_cache_api_groups_count Number of cluster cache api groups
# TYPE argocd_cluster_cache_api_groups_count gauge
argocd_cluster_cache_api_groups_count{cluster="https://foo.bar:6443"} 43

PS: This is part of https://github.com/argoproj/gitops-engine/pull/350

[sonarcloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codecov[bot]]

Codecov Report

Base: 55.38% // Head: 55.48% // Increases project coverage by +0.09% :tada:

Coverage data is based on head (ac85096) compared to base (c036d3f). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #458 +/- ## ========================================== + Coverage 55.38% 55.48% +0.09% ========================================== Files 41 41 Lines 4478 4506 +28 ========================================== + Hits 2480 2500 +20 - Misses 1807 1813 +6 - Partials 191 193 +2 ``` | [Impacted Files](https://codecov.io/gh/argoproj/gitops-engine/pull/458?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=argoproj) | Coverage Δ | | |---|---|---| | [pkg/cache/cluster.go](https://codecov.io/gh/argoproj/gitops-engine/pull/458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=argoproj#diff-cGtnL2NhY2hlL2NsdXN0ZXIuZ28=) | `54.99% <100.00%> (+0.13%)` | :arrow_up: | | [pkg/diff/diff.go](https://codecov.io/gh/argoproj/gitops-engine/pull/458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=argoproj#diff-cGtnL2RpZmYvZGlmZi5nbw==) | `61.09% <0.00%> (+0.33%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=argoproj). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=argoproj)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[alswl]

ping @terrytangyuan

[alswl]

@alexmt Hi, do you have time to look at this Pull Request as well?

[zachaller]

We should also add docs to https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/

[alswl]

@zachaller

We should also add docs to https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/

Yes, I created a related pull request(draft) for ArgoCD, It should be replaced by new version gitops-engine after merge.

• discuss: https://github.com/argoproj/argo-cd/issues/11157
• pr: https://github.com/argoproj/argo-cd/pull/11156

[sonarcloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

[alswl]

Friendly ping with @zachaller and @alexmt .