search

kcp-dev / contrib-tmc

An experimental add-on readding some Kubernetes compute APIs and impement transparent multi-cluster scheduling
Apache License 2.0
5 stars 3 forks source link

bug: Syncer not working as expected #64

Open yogesh-reddy opened 1 year ago

yogesh-reddy commented 1 year ago

Describe the bug

i could see error in syncer pod logs kubectl logs kcp-syncer-dp-8568db4-1a1jng80-6b6f958b48-7kcxn | grep E0

E0111 18:02:32.387938       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0111 18:02:32.388423       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0111 18:02:32.388707       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0111 18:02:32.426879       1 controller.go:240] failed to sync "dp-8568db4": Operation cannot be fulfilled on synctargets.workload.kcp.io "dp-8568db4": the object has been modified; please apply your changes to the latest version and try again
E0111 18:02:33.374780       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource

versions: my kcp version : kcp version v1.24.3+kcp-v0.10.0-429-g8568db4b5eb633 syncer image used : ghcr.io/kcp-dev/kcp/syncer:8568db4

Steps To Reproduce

kcp version : kcp version v1.24.3+kcp-v0.10.0-429-g8568db4b5eb633 syncer image used : ghcr.io/kcp-dev/kcp/syncer:8568db4 user this and try to add workload cluster

Expected Behaviour

the syncer should be able to work succesfully

Additional Context

No response

stevekuznetsov commented 1 year ago

cc @davidfestal

davidfestal commented 1 year ago

these lines

E0111 18:02:32.387938       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0111 18:02:32.388423       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0111 18:02:32.388707       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource

are not very helpful and possibly not the cause of the error at all.

Could you provide more logs of the Syncer ? Could you provide more information about what doesn't work ? And more details about the steps you use to create:

yogesh-reddy commented 1 year ago

@davidfestal i hope this is the error E0111 18:02:32.426879 1 controller.go:240] failed to sync "dp-8568db4": Operation cannot be fulfilled on synctargets.workload.kcp.io "dp-8568db4": the object has been modified; please apply your changes to the latest version and try again i tried to run kubectl kcp workload sync command and then applied he yaml generated on my kind cluster. later when i tried to apply deployments it could not find the resource .. the deployments are not registered with kcp so went ahead and checked the syncer logs and i could see the above log. let me know if i am doing it wrong

davidfestal commented 1 year ago

@davidfestal i hope this is the error E0111 18:02:32.426879 1 controller.go:240] failed to sync "dp-8568db4": Operation cannot be fulfilled on synctargets.workload.kcp.io "dp-8568db4": the object has been modified; please apply your changes to the latest version and try again

I would be surprised it to be the root cause.

when i tried to apply deployments it could not find the resource

That would be more the cause of the problem I assume. Can you provide more details about the steps you used ?

yogesh-reddy commented 1 year ago

nothing much .. started kcp (versioned mentioned above) generated workload kubectl kcp workload sync dp-8568db4 --syncer-image ghcr.io/kcp-dev/kcp/syncer:8568db4 -o syncer-8568db4-kind-main.yaml yaml is generated applied that yaml and saw those error in logs

davidfestal commented 1 year ago

But does the Syncer POD crashes or not ?

yogesh-reddy commented 1 year ago

its running kcp-syncer-dp-8568db4-1a1jng80-6b6f958b48-7kcxn 1/1 Running 0 25h on kcp control palne sync target is present

❯ kubectl get synctarget NAME AGE dp-8568db4 25h

yogesh-reddy commented 1 year ago

FYI just restated the pod the total logs

kubectl logs kcp-syncer-dp-8568db4-1a1jng80-6b6f958b48-njmv2

0112 19:06:03.955087       1 syncer.go:83] "syncing" resource-types=[]
I0112 19:06:03.963238       1 syncer.go:85] "starting syncer" syncTarget.workspace="root" syncTarget.name="dp-8568db4"
I0112 19:06:03.965074       1 syncer.go:113] "attempting to retrieve the Syncer virtual workspace URL" syncTarget.workspace="root" syncTarget.name="dp-8568db4"
I0112 19:06:04.264370       1 syncer.go:261] "Creating spec syncer" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz"
I0112 19:06:04.265325       1 namespace_downstream_controller.go:166] "Set up downstream namespace informer" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-workload-syncer-namespace-downstream"
I0112 19:06:04.268615       1 syncer.go:279] "Creating status syncer" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz"
I0112 19:06:04.269022       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=secrets"
I0112 19:06:04.269398       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=secrets"
I0112 19:06:04.269506       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=namespaces"
I0112 19:06:04.269602       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=namespaces"
I0112 19:06:04.281038       1 controller.go:196] "queueing SyncTarget" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:04.370459       1 status_controller.go:171] "Starting syncer workers" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-workload-syncer-status"
I0112 19:06:04.370566       1 apiimporter.go:174] "Starting API Importer" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer"
I0112 19:06:04.370566       1 spec_controller.go:305] "Starting syncer workers" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-workload-syncer-spec"
I0112 19:06:04.370808       1 shared_informer.go:255] Waiting for caches to sync for kcp-ddsif-gvr-source
I0112 19:06:04.370814       1 shared_informer.go:255] Waiting for caches to sync for kcp-ddsif-gvr-source
I0112 19:06:04.370857       1 shared_informer.go:262] Caches are synced for kcp-ddsif-gvr-source
I0112 19:06:04.370873       1 shared_informer.go:262] Caches are synced for kcp-ddsif-gvr-source
I0112 19:06:04.370873       1 apiimporter.go:227] "Importing APIs" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" resourcesToImport=[deployments.apps ingresses.networking.k8s.io services.]
I0112 19:06:04.370928       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=configmaps"
I0112 19:06:04.371042       1 shared_informer.go:255] Waiting for caches to sync for kcp-ddsif-gvr-source
I0112 19:06:04.371131       1 shared_informer.go:262] Caches are synced for kcp-ddsif-gvr-source
I0112 19:06:04.371184       1 controllermanager.go:117] "Starting controller manager" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="syncer-controller-manager-upstream-upsyncer"
I0112 19:06:04.371254       1 controllermanager.go:117] "Starting controller manager" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="syncer-controller-manager-upstream-syncer"
I0112 19:06:04.371448       1 controllermanager.go:117] "Starting controller manager" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="syncer-controller-manager-downstream-syncer"
I0112 19:06:04.371507       1 controller.go:208] "Starting controller" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller"
I0112 19:06:04.371523       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=configmaps"
I0112 19:06:04.371131       1 namespace_downstream_controller.go:204] "Starting controller" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-workload-syncer-namespace-downstream"
I0112 19:06:04.371724       1 controller.go:233] "processing key" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:04.382603       1 controller.go:196] "queueing SyncTarget" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:04.409541       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=services"
I0112 19:06:04.409605       1 informer.go:314] "adding dynamic informer for gvr" gvr="/v1, Resource=services"
I0112 19:06:04.409654       1 informer.go:314] "adding dynamic informer for gvr" gvr="networking.k8s.io/v1, Resource=ingresses"
I0112 19:06:04.409714       1 informer.go:314] "adding dynamic informer for gvr" gvr="apps/v1, Resource=deployments"
I0112 19:06:04.409541       1 controller.go:233] "processing key" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:04.410239       1 informer.go:314] "adding dynamic informer for gvr" gvr="networking.k8s.io/v1, Resource=ingresses"
I0112 19:06:04.410867       1 informer.go:314] "adding dynamic informer for gvr" gvr="apps/v1, Resource=deployments"
I0112 19:06:04.411653       1 controllermanager.go:152] "waiting for the informer to be synced before starting controller" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" gvr="/v1, Resource=services" controller="syncer-endpoint-controller"
I0112 19:06:04.414597       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=services"
W0112 19:06:04.415599       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:04.415675       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:04.415741       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:04.415614       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:04.415786       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:04.415803       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
I0112 19:06:04.422794       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="networking.k8s.io/v1, Resource=ingresses"
I0112 19:06:04.425895       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="apps/v1, Resource=deployments"
I0112 19:06:04.428823       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=configmaps"
I0112 19:06:04.433551       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=secrets"
W0112 19:06:05.290755       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:05.291145       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:05.495052       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:05.495188       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:05.584174       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:05.584252       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
I0112 19:06:05.622775       1 request.go:601] Waited for 1.046215s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
I0112 19:06:07.222433       1 request.go:601] Waited for 1.0454483s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/discovery.k8s.io/v1?timeout=32s
I0112 19:06:07.675144       1 discovery.go:186] "processing discovery" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" groupVersion="v1" resource="services" kind="Service" crd="services.core"
I0112 19:06:07.683124       1 discovery.go:186] "processing discovery" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" groupVersion="apps/v1" resource="deployments" kind="Deployment" crd="deployments.apps"
I0112 19:06:07.692123       1 discovery.go:186] "processing discovery" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" groupVersion="networking.k8s.io/v1" resource="ingresses" kind="Ingress" crd="ingresses.networking.k8s.io"
I0112 19:06:07.701672       1 apiimporter.go:271] "updating APIResourceImport" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" group="apps" version="v1" resource="deployments" apiResourceImport="deployments.dp-8568db4.v1.apps"
I0112 19:06:07.754085       1 apiimporter.go:271] "updating APIResourceImport" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" group="networking.k8s.io" version="v1" resource="ingresses" apiResourceImport="ingresses.dp-8568db4.v1.networking.k8s.io"
I0112 19:06:07.762320       1 apiimporter.go:271] "updating APIResourceImport" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" resources=[] reconciler="api-importer" group="" version="v1" resource="services" apiResourceImport="services.dp-8568db4.v1.core"
W0112 19:06:07.825158       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:07.825254       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:07.953574       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:07.953624       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:08.695425       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:08.695539       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:12.862075       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:12.862161       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:12.899413       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:12.899497       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:13.298070       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:13.298172       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:21.304522       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:21.304607       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:21.363398       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:21.363483       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
I0112 19:06:24.391125       1 controller.go:196] "queueing SyncTarget" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:24.391317       1 controller.go:233] "processing key" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:24.402117       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="apps/v1, Resource=deployments"
I0112 19:06:24.406742       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=secrets"
I0112 19:06:24.411319       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=configmaps"
I0112 19:06:24.415566       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=services"
I0112 19:06:24.419516       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="networking.k8s.io/v1, Resource=ingresses"
W0112 19:06:25.134675       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:25.134867       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:34.917471       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:34.917531       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
W0112 19:06:42.098530       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:42.098687       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
I0112 19:06:44.363748       1 controller.go:196] "queueing SyncTarget" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:44.363929       1 controller.go:233] "processing key" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4"
I0112 19:06:44.370392       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=configmaps"
I0112 19:06:44.374857       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=secrets"
I0112 19:06:44.378804       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="/v1, Resource=services"
I0112 19:06:44.383200       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="networking.k8s.io/v1, Resource=ingresses"
I0112 19:06:44.387301       1 controller.go:420] "Informer is started already" syncTarget.workspace="root" syncTarget.name="dp-8568db4" syncTarget.key="cKK89Yvux7z7kBVBj3oVPIY9VBlLrL97v8mjtz" reconciler="kcp-syncer-synctarget-gvrsource-controller" key="dp-8568db4" gvr="apps/v1, Resource=deployments"
W0112 19:06:46.572773       1 reflector.go:324] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: failed to list *unstructured.Unstructured: the server could not find the requested resource
E0112 19:06:46.572889       1 reflector.go:138] k8s.io/client-go@v0.24.4/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource
smalleni commented 1 year ago

I'm not sure if my issue is related but I also have issues getting syncer to run.

KCP version

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3+kcp-v0.10.0", GitCommit:"25254541", GitTreeState:"clean", BuildDate:"2022-12-13T17:11:33Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/arm64"}

Error in syncer pod on the physical kind cluster

      Message:   I0116 16:21:30.652098       1 syncer.go:83] "syncing" resource-types=[]
I0116 16:21:30.673366       1 syncer.go:85] "starting syncer" syncTarget.workspace="root" syncTarget.name="kind"
I0116 16:21:30.678380       1 syncer.go:113] "attempting to retrieve the Syncer virtual workspace URL" syncTarget.workspace="root" syncTarget.name="kind"
E0116 16:21:30.749586       1 run.go:74] "command failed" err="synctargets.workload.kcp.io \"kind\" is forbidden: User \"system:serviceaccount:default:kcp-syncer-kind-29c7eyul\" cannot get resource \"synctargets\" in API group \"workload.kcp.io\" at the cluster scope: access denied"

Manifest generated by the command for ClusterRole

kubectl kcp workload sync kind --syncer-image ghcr.io/kcp-dev/kcp/syncer:9358474@sha256:da59b1438fc38cddea8eed3ecc00ea35cb60de9780d38f83714908f578da6e3d -o syncer-kind-main.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kcp-syncer-kind-29c7eyul
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - "create"
  - "get"
  - "list"
  - "watch"
  - "delete"
- apiGroups:
  - "apiextensions.k8s.io"
  resources:
  - customresourcedefinitions
  verbs:
  - "get"
  - "watch"
  - "list"
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - services
  verbs:
  - "*"
- apiGroups:
  - "apps"
  resources:
  - deployments
  verbs:
  - "*"
- apiGroups:
  - "networking.k8s.io"
  resources:
  - ingresses
  verbs:
  - "*"

The syncer pod keeps crashlooping.

yogesh-reddy commented 1 year ago

I'm not sure if my issue is related but I also have issues getting syncer to run.

KCP version

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3+kcp-v0.10.0", GitCommit:"25254541", GitTreeState:"clean", BuildDate:"2022-12-13T17:11:33Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/arm64"}

Error in syncer pod on the physical kind cluster

      Message:   I0116 16:21:30.652098       1 syncer.go:83] "syncing" resource-types=[]
I0116 16:21:30.673366       1 syncer.go:85] "starting syncer" syncTarget.workspace="root" syncTarget.name="kind"
I0116 16:21:30.678380       1 syncer.go:113] "attempting to retrieve the Syncer virtual workspace URL" syncTarget.workspace="root" syncTarget.name="kind"
E0116 16:21:30.749586       1 run.go:74] "command failed" err="synctargets.workload.kcp.io \"kind\" is forbidden: User \"system:serviceaccount:default:kcp-syncer-kind-29c7eyul\" cannot get resource \"synctargets\" in API group \"workload.kcp.io\" at the cluster scope: access denied"

Manifest generated by the command for ClusterRole

kubectl kcp workload sync kind --syncer-image ghcr.io/kcp-dev/kcp/syncer:9358474@sha256:da59b1438fc38cddea8eed3ecc00ea35cb60de9780d38f83714908f578da6e3d -o syncer-kind-main.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kcp-syncer-kind-29c7eyul
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - "create"
  - "get"
  - "list"
  - "watch"
  - "delete"
- apiGroups:
  - "apiextensions.k8s.io"
  resources:
  - customresourcedefinitions
  verbs:
  - "get"
  - "watch"
  - "list"
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - services
  verbs:
  - "*"
- apiGroups:
  - "apps"
  resources:
  - deployments
  verbs:
  - "*"
- apiGroups:
  - "networking.k8s.io"
  resources:
  - ingresses
  verbs:
  - "*"

The syncer pod keeps crashlooping. @smalleni i had this issue when i was using a different version of syncer for a given kcp version. the pod crash status went by but then the above mentioned issue is still exsiting.

smalleni commented 1 year ago

workload.kcp.io

My issue seems to be from the fact that the ClusterRole generated does not include the apiGroup workload.kcp.io, is that what you are also experiencing?

smalleni commented 1 year ago

workload.kcp.io

My issue seems to be from the fact that the ClusterRole generated does not include the apiGroup workload.kcp.io, is that what you are also experiencing?

Never mind, using the syncer image with the same tag as the KCP version resulted in the syncer pod running, but i see similar errors as @yogesh-reddy in the syncer pod logs and am unable to create a deployment on the physical cluster through kcp

kasturinarra commented 1 year ago

@smalleni to be able to create a deployment, you would need to do as below. There have been some changes with respect to the workflow in v0.10.0

  1. Run the command kubectl kcp bind compute ""
  2. Now try to create deployment and it should work.

p.s: If you are trying to add deployment as a normal user you might hit issue described in here https://github.com/kcp-dev/kcp/issues/2597 follow steps in here https://github.com/kcp-dev/kcp/issues/2597#issuecomment-1380862841 to workaround. If you are trying to add deployment as an admin user may be things will be smooth for you

pdettori commented 1 year ago

I have ran in the same issues ... this new workflow should be documented (e.g. in https://docs.kcp.io/kcp/main/ )

vishnuchalla commented 1 year ago

@smalleni to be able to create a deployment, you would need to do as below. There have been some changes with respect to the workflow in v0.10.0

  1. Run the command kubectl kcp bind compute ""
  2. Now try to create deployment and it should work.

p.s: If you are trying to add deployment as a normal user you might hit issue described in here kcp-dev/kcp#2597 follow steps in here #2597 (comment) to workaround. If you are trying to add deployment as an admin user may be things will be smooth for you

Thanks this resolution worked for me.

mjudeikis commented 9 months ago

/transfer-issue contrib-tmc