GoogleCloudPlatform / metacontroller

Lightweight Kubernetes controllers as a service
https://metacontroller.app/
Apache License 2.0
792 stars 111 forks source link

CRD updates require restart of metacontroller #181

Open cliffpracht opened 5 years ago

cliffpracht commented 5 years ago

Hi there,

We've noticed an unexpected behavior with the metacontroller - When you update the actual CRD of the parent, Metacontroller is no longer able to update the status of the parent resource. Here is log below - we updated the CRD at 15:40:50.400125, it reported the resourceversion changed, followed by inability to update the status:

W0806 15:40:50.400125       1 reflector.go:341] metacontroller.app/dynamic/informer/factory.go:111: watch of *unstructured.Unstructured ended with: unexpected object: &{map[code:410 kind:Status apiVersion:v1 metadata:map[] status:Failure message:too old resource version: 323493 (323653) reason:Gone]}
I0806 15:40:51.401780       1 reflector.go:240] Listing and watching *unstructured.Unstructured from metacontroller.app/dynamic/informer/factory.go:111
I0806 15:40:51.420169       1 controller.go:406] sync CustomResourceExample demo-not-injected/service
I0806 15:40:51.420226       1 controller.go:406] sync CustomResourceExample demo-injected/service
I0806 15:42:02.519743       1 controller.go:406] sync CustomResourceExample demo-injected/service
E0806 15:42:02.841345       1 controller.go:223] failed to sync CustomResourceExample "demo-injected/service": can't update status for CustomResourceExample demo-injected/service: the server could not find the requested resource

We are using the compositecontroller functionality of the metacontroller where we have a single parent with multiple children.

To fix issue we need to restart the metacontroller.

Is this expected behavior? If so, what is the recommended method to update CRDs , should we change the version and then spin up a compositecontroller to look at the new version?

Typically, we do not update the CRDs, but we are hitting this case when we deploy the CRD, which may come after the Metacontroller is installed/CompositeController is created.

Thanks!