search

pusher / faros

Faros is a CRD based GitOps controller
Apache License 2.0
99 stars 15 forks source link

v0.4.0-rc3 // error syncing child XXX : unable to patch object: the server rejected our request for an unknown reason (patch deployments.apps api) #124

Closed looztra closed 5 years ago

looztra commented 5 years ago

Context

What?

We upgraded from v0.4.0-rc1 to v0.4.0-rc3 (because we want to use git over https with token instead of git to avoid ssh known_hosts_problems with github) and we are now facing with errors like :

faros-controller-manager-0 manager 2019/04/05 09:34:45 Getting repository 'https://github.com/company/k8s-apps-descriptors.git'
faros-controller-manager-0 manager 2019/04/05 09:34:45 Checking out 'master'
faros-controller-manager-0 manager 2019/04/05 09:34:50 error syncing child apps-demo/deployment-api: error updating child Deployment api: unable to update child: unable to update child resource: error applying update: unable to patch object: the server rejected our request for an unknown reason (patch deployments.apps api)
faros-controller-manager-0 manager 2019/04/05 09:34:56 error syncing child apps-demo/deployment-api: error updating child Deployment api: unable to update child: unable to update child resource: error applying update: unable to patch object: the server rejected our request for an unknown reason (patch deployments.apps api)

Samples of the descriptors we use are available at : https://gist.github.com/looztra/dc48d8031f895c53e3ae49cb227c796c

We have tried to reset everything (faros down, all ressources deleted, then faros up) and we observed that one of our resources (the deployment in the previous log) generated this kind of error even the first time it is applied by faros (but the deployment IS applied this first time).

And when changes are to be applied to several resources after the init (after first faros apply), then nothing happens this time (no resource that should be changed are changed) and we get these errors for all the resources that should have been applied.

Any suggestion?

JoelSpeed commented 5 years ago

Hi @looztra, I'm planning to release a v0.4.0-rc4 some time this morning which will fix your issues (I think 🙈)

The unuseful the server rejected our request for an unknown reason errors have been fixed by https://github.com/pusher/faros/pull/117 and once that is in, you'll probably get an error on GKE saying dry run not yet supported. Use the new --server-dry-run=false https://github.com/pusher/faros/pull/121 flag to disable dry run and you'll be good again

We've only recently started using Faros on GKE and never used Faros on a 1.12 cluster before this. We have now set up testing for 1.11, 1.12 and 1.13 in our CI suite to try and prevent these issues in the future.

Apologies for the inconvenience this has caused you

JoelSpeed commented 5 years ago

The RC4 image is now available

looztra commented 5 years ago

Hi @JoelSpeed , thank you very much for your super quick answer!

We agreed by using RC versions that we may face this kind of issues, so that's not a problem at all!

We will try the new version in the wake of this message!

looztra commented 5 years ago

Works fine with v0.4.0-rc4 :

Thank you!

JoelSpeed commented 5 years ago

Awesome! Thanks for circling back 😄