kcp-dev / contrib-tmc

An experimental add-on readding some Kubernetes compute APIs and impement transparent multi-cluster scheduling
Apache License 2.0
5 stars 3 forks source link

:seedling: TMC is live! #2

Closed mjudeikis closed 11 months ago

mjudeikis commented 12 months ago

In this PR:

Important:

mjudeikis commented 12 months ago

Before we open this to the world I would like to get e2e running as currently they are failing. But failures looks like more of the migration, not code.

fgiloux commented 12 months ago

I have tried to run locally and the syncer is able to connect to kcp and to update the status. When trying to bind the kubernetes APIExport and to create a deployment it is working in root:kubernetes but not in another workspace: kvdk2spgmbix:test-sync. I get the following error:

kubectl kcp bind compute root:compute:kubernetes
placement placement-jsym7slt created.
error: placement "placement-jsym7slt" not ready: placement is not scheduled: No selected location is scheduled

It is possible that I am simply missing a step.

There are a few minor things in Makefile:

I will try to look at the code changes when time allows.

mjudeikis commented 12 months ago

When trying to bind the kubernetes APIExport and to create a deployment it is working in root:kubernetes but not in another workspace: kvdk2spgmbix:test-sync. I get the following error:

I think it created default placement, and if there are no default location (no synctarget present) it will fail. This is what I found in the code before, and didn't changed.

Did you had synctarget in the test-sync workspace?

fgiloux commented 11 months ago

Log forwarding is not working but that's something we can look at later on:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
kuard-587f7fc9cc-ptdst   1/1     Running   0          46s
[fgiloux@fgiloux contrib-tmc]$ kubectl logs kuard-587f7fc9cc-ptdst 
Error from server (NotFound): pods "kuard-587f7fc9cc-ptdst" not found
fgiloux commented 11 months ago

I am still facing this issue. I will try to troubleshoot it tomorrow

mjudeikis commented 11 months ago

Yes! Now I remember:

// TODO(MJ): currently this disables the cross workspace placements. This is due to
    // fact indexers are build ontop of logicalcluster paths and we use here the 'readable' paths
    // if len(placement.Spec.LocationWorkspace) > 0 {
    //  locationWorkspace = logicalcluster.NewPath(placement.Spec.LocationWorkspace)
    // } else {
    locationWorkspace = logicalcluster.From(placement).Path()
    // }

Will create an issue to follow-up with it

mjudeikis commented 11 months ago

Currently, it works for basic deploy container to pcluster. No sharding, no cross-workspace locations/placements,no pod logs. All of these things would have to come with VM refactoring. But there are big limitations on the current approach due to how KCP setups informers (when in the lifecycle) and if you extend you can't use same patterns. In addition, the caching server is not working for 3rd party resource as it expects it to be using workspaces/apiexports.

I think we need to merge this, and start working on VW implementation where we would be deploying ontop of KCP fully. As feel like we just need rip the cord and move on with this.