feature: Allow customizing the limited set of resource types that can be upsynced

[davidfestal]

For now only a fixed set of resources can be upsynced: pods, endpoints, persistentvolumes.

Limiting what can be upsynced from the physical cluster is quite important. We don't want to open the door to any kind of respources. OTOH we could think about a mechanism that would allow defining, on the SyncTarget status (in the status.syncedresources ? or in any other field) whether each resource can be upsynced or synced. A sort of fine-grained filter of what is allowed (for both syncing and upsyncing) for every resource type exposed to the Syncer. Having that in the SyncTarget status would allow the Syncer itself to follow the same rules.

The question of how to define such rules is entirely open, but let's make the whole as simple as possible at start.

[pdettori]

Here is a use case where I see the need to up-sync a secret: Syncing of Crossplane Claims use case: This is a similar use case to the one demoed by @sttts at Kubecon NA 2022 for Kube-bind. In this use case, Crossplane is installed in the physical cluster and Crossplane claims CRDs are imported in the kcp workspace. Crossplane claims typically create connection secrets, which are created in the backend cluster. But to give access to the resources created by those claims, the developer need to have access to the connection secret. Therefore, the connection secret should be up synced to the workspace where the claim lives. I created a PoC for a controller that runs downstream and for each Crossplane connection secret finds the owner (the down-synced claim) and can add the UpsSync label to the connection secret. I was hoping to use this PR to now close the loop and be able to up sync the connection secret. But it looks like with the current restriction that will not be possible unless I create my own fork for the syncer.

[mamachanko]

I am also interested in finer control over the upsyncable resources. The use-case of interest is largely identical to the scenario described by @pdettori.

Regardless of the approach, an interesting number of scenarios would be addressed if Secrets would be upsyncable. On the other hand, these are also one of the most security-critical resources.

[mamachanko]

a mechanism that would allow defining, on the SyncTarget status (in the status.syncedresources ? or in any other field) whether each resource can be upsynced or synced.

@davidfestal Just trying to see if I correctly follow your thinking, by thinking out loud:

1. somewhere a user could mark a resource as upsyncable, e.g. kubectl kcp workload sync ... --upsyncableResources ...
2. the resulting SyncTarget would record the upsyncable resources in its .status
3. when the syncer encounters a resource which is labelled as "upsync me to sync target xyz", then it would consult SyncTarget.status and police the upsyncing of the resource accordingly
   • if allowed/accepted, then upsync to the sync target
   • if forbidden/incompatible, then just log

[mjudeikis]

/transfer-issue contrib-tmc