Pods not synced back from physical cluster to kcp

[lburgazzoli]

Describe the bug Pods are not synced from the physical cluster back to kcp

To Reproduce Started kcp from main with

kcp start \
    --auto-publish-apis=true \
    --pull-mode=false \
    --push-mode=true \
    --run-controllers=true \
    --resources-to-sync=deployments.apps,pods

kcp starts and reports:

syncer.go:141] "Set up informer" direction=kcpToPhysicalCluster clusterName="connectors" logicalCluster="admin" gvr="/v1, Resource=namespaces"
syncer.go:141] "Set up informer" direction=kcpToPhysicalCluster clusterName="connectors" logicalCluster="admin" gvr="/v1, Resource=pods"
syncer.go:141] "Set up informer" direction=kcpToPhysicalCluster clusterName="connectors" logicalCluster="admin" gvr="apps/v1, Resource=deployments"
syncer.go:141] "Set up informer" direction=physicalClusterToKcp clusterName="connectors" logicalCluster="admin" gvr="/v1, Resource=namespaces"
syncer.go:141] "Set up informer" direction=physicalClusterToKcp clusterName="connectors" logicalCluster="admin" gvr="/v1, Resource=pods"
syncer.go:141] "Set up informer" direction=physicalClusterToKcp clusterName="connectors" logicalCluster="admin" gvr="apps/v1, Resource=deployments"

Then register a physical cluster named my-cluster using kind Then create a deployment

kubectl apply -f contrib/examples/deployment.yaml
kubectl label --overwrite deployment example kcp.dev/cluster=my-cluster

At this point, the deployment gets synchronized with the physical cluster to a namespace created by kcp and the related pods starts:

➜ kubectl get deployments.apps,pods -n kcpb0a2aacd123de54bf53be16aa7fdf1a9262aec753e1a27717df04e7b
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/example   1/1     1            1           15m

NAME                           READY   STATUS    RESTARTS   AGE
pod/example-5cbdcd89b7-5dwn5   1/1     Running   0          15m

However when looking at kcp, I see that the deployment is properly synced and reported as running but I don't see any pod being synced back:

➜ kubectl get deployments.apps,pods -n my-ns
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/example   1/1     1            1           16m

Expected behavior Pods should be synced to kcp

[astefanutti]

I've added some details about the use cases, and a possible fix in #520.

[sttts]

IMO --resources-to-sync is strickly for kcp->pcluster. It is not meant for any kind of two-way sync. If we want that, we need an API to specify those objects.

Also it is not clear that we actually want to persist pods in kcp. There are different in nature from the resources-to-sync resources because the spec is owned by the pcluster. We have talked about a proxy mechanism the syncer would provide and the kcp apiserver could use to return pods, basically tunneled through from the pcluster.

[astefanutti]

The key point here IMO is how transparent that is for external controllers / clients. Ideally, that should be completely transparent to them.

[sttts]

I totally agree with the use-case. Non-kcp-native, non-federation-aware controllers expect pods. So if we want a kube-native experience we have to make pods visible. My point is how we do that and how we instruct the system to do it.

[sttts]

As background: pods have high cardinality and are heavy on updates. Making them visible via the syncer might quickly exhaust our load budget for individual workspaces.

And all this only talks about the 1:1 (non-federation) case. The moment two pclusters are involved, even naming of pods is not guaranteed to be unique. Imagine during a rolling migration to a new pcluster (eventually) we have two sources of pods.

[sttts]

To sum up: not saying we don't want this. We want and need pod visiblity, just slowing down a bit to make a plan how we want to implement it :)

[astefanutti]

@sttts thanks, everything you've said makes total sense! Better giving it more thinking, and doing it right 👍🏼.

[ncdc]

Related: #25