search

2gis / k8s-handle

Easy CI/CD for Kubernetes clusters with python and jinja2
Apache License 2.0
170 stars 30 forks source link

k8s-handle fails on serialized deploy (CRD -> Object) #111

Open ghost opened 4 years ago

ghost commented 4 years ago

CRD defined object deploy fails even in right sequence serialized case:

  1. Deploy CRD
  2. Deploy custom object

config.yaml:

  - template: "zalando-postgres-operator/OperatorConfigurationCustomResourceDefinition.yaml.j2"
  - template: "zalando-postgres-operator/OperatorConfiguration.yaml.j2"

Log:

2020-01-09 07:24:52 ERROR:k8s_handle:RuntimeError: No valid plural name of resource definition discovered
rvadim commented 4 years ago

Hi, can you provide full example here, so we can reproduce problem?

furiousassault commented 4 years ago

@rvadim, I think we see how the issue can be reproduced.

The problem is, when we have a CRD and the objects of that CRD in the same deployment, it's highly likely will be failed due to some time gap between CRD accept by the Apiserver and the moment when it can handle the objects of that CRD.

Sort of my bad, when I implemented this functionality, I haven't checked and handled this case: deployed CRD and objects in different deployments and didn't want to make logic more complex. We hadn't used deployments described by @kichay at that moment, so it wasn't a high priority issue.

To avoid that, we should implement a mechanism of CRD deployment which listens the events until the event that confirms CRD readiness appears (or timeout expires), and continue deployment or fail after that.

I can take this issue when I'll have some free time, or you can take it, if you want.