feature: deployments couldn’t goes back to the origin replicas count in kcp by syncer reconciler after manual change the deployment in pcluster

[Phaow]

Describe the bug

Deployments couldn’t goes back to the origin replicas count in kcp by syncer reconciler after manual change the deployment in pcluster

Steps To Reproduce

1. Create workload sync and generate syncer resources manifests in current workspace and apply syncer resources on pcluster and wait for synctarget become ready
2. Create deployment from kcp by running the command below

   $ kubectl create deployment kuard --image gcr.io/kuar-demo/kuard-amd64:blue --dry-run=client -o yaml > kuard.yaml
   $ kubectl apply -f kuard.yaml

3. Verify that deployments have been created by running the command below

   $ kubectl get deployments

4. Now login to the pcluster and edit the deployment kuard to set replicas to 10 by running the command below

   $ oc scale deployment/kuard --replicas=0

5. Since kcp is the source of truth validate that it reconciles back and deployment goes back to what ever replica count(should be 1) that has been set in kcp.

Actual result The deployment couldn’t goes back to the origin replicas count (should be 1) in kcp.

# After login to pcluster and edit the deployment kuard to set replicas to 10

# The deployment info on Pcluster
$ oc -n kcp-2t1y1qx7f9wa get deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
e2e-dep-rsod8vx2   10/10   10           10          6m44s

# The deployment info in kcp
$k get -n e2e-ns-kcp-syncer-cb84c deploy
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
e2e-dep-rsod8vx2   10/1    10           10          6m51s

Expected Behaviour

At Step 5 The deployment goes back to the origin replicas count in kcp.

Additional Context

Version: v0.10.0 (v0.9.1 also has the same issue)

In the syncer logs, we could see the error info as follows:

E0130 14:19:51.175494       1 status_controller.go:167] kcp-workload-syncer-status failed to sync {{"apps" "v1" "deployments"} "kcp-4194ng45jm8g/kuard"}, err: Operation cannot be fulfilled on deployments:1f2dee4bb55028e24cfd892fed90469b6a9159af17a79095c848b4280080354c.apps "kuard": the resource has been modified in the meantime

[kasturinarra]

@Phaow as per the discussion in the thread below it looks like 'The only downstream event which triggers a new syncing-down of an upstream resource is when a downstream resource is deleted.' let us update our case to delete the downstream resource and see if it gets synced down. https://redhat-internal.slack.com/archives/C020QAF9D46/p1675170468647729?thread_ts=1675159108.057229&cid=C020QAF9D46

And let us change this as a feature for now, more context in here https://redhat-internal.slack.com/archives/C020QAF9D46/p1675185026457319?thread_ts=1675184496.085559&cid=C020QAF9D46

[Phaow]

Update the case steps and code in https://github.com/kcp-dev/kcp-tests/pull/29 .

[Phaow]

/label kind/feature

[openshift-ci[bot]]

@Phaow: The label(s) /label kind/feature cannot be applied. These labels are supported: platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved, downstream-change-needed, approved, backport-risk-assessed, bugzilla/valid-bug, cherry-pick-approved, jira/valid-bug, staff-eng-approved. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/kcp-dev/contrib-tmc/issues/59): >/label kind/feature Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[Phaow]

kind /feature

[mjudeikis]

/transfer-issue contrib-tmc