✨Bypass vc-syncing if tenant has own scheduler and dedicated node.

[weiling61]

What this PR does / why we need it: For certain tenants, they may be assigned with dedicated nodes and can schedule workload (pod) to these nodes with tenant scheduler. We would like to introduce a new label: "tenancy.x-k8s.io/tenant-scheduler" to let vc-syncer to be aware of the situation and skip the syncing.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes # N/A

[christopherhein]

I am good with this since it's opt-in @Fei-Guo @wondywang would either of you be able to check this out as well?

[Fei-Guo]

Can you explain more about the background? Are you directly registering nodes to tenant control plane?

In the virtual cluster model, even if tenant wants to use dedicated nodes, the nodes should be registered to super cluster and exposed to tenant control plane using vNodes.

If you directly register node to tenant control plane, I'd rather you just introduce a generic feature to mark vPod not to be synchronized.

[weiling61]

@Fei-Guo "Can you explain more about the background? Are you directly registering nodes to tenant control plane?" [answer] Yes I directly registered nodes for some Virtual Clusters and added a scheduler for each virtual cluster.

"In the virtual cluster model, even if tenant wants to use dedicated nodes, the nodes should be registered to super cluster and exposed to tenant control plane using vNodes." [answer] This is still an experimental feature. Directly registering node to virtual cluster brings other advantages. In near future, we may propose to block other resources from syncing to super cluster.

"If you directly register node to tenant control plane, I'd rather you just introduce a generic feature to mark vPod not to be synchronized." [answer] It is not just about vPod. I may need use this feature gate and label to block other resources from syncing to super cluster. We may propose a more generic and comprehensive solution once I complete prototype.

[Fei-Guo]

@weiling61

You can choose label name like "tenancy.x-k8s.io/ignore-sync=true" and just make a feature to prevent synchronizing any objects. The feature name "TenantScheduler" does not make much sense for other non-Pod objects anyway.

[weiling61]

@Fei-Guo Change the new label suggested in your comments

[Fei-Guo]

/approve /lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: christopherhein, Fei-Guo, weiling61

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[virtualcluster/OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/OWNERS)~~ [Fei-Guo,christopherhein] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment