search

AmitKumarDas / metac

It is metacontroller and more
Apache License 2.0
57 stars 16 forks source link

feat(gctl): #142 add indefinite retry option during metac startup #143

Closed AmitKumarDas closed 4 years ago

AmitKumarDas commented 4 years ago

This commit closes #142

A new boolean flag named retry-indefinitely-to-start is added that lets metac to retry indefinitely to start all the configured generic controllers. When this flag is set to true, retries happen even when one or more configured kubernetes custom resource definitions are not available at the cluster. Once these custom definitions are available & discovered by metac, this retry stops completely.

When this flag is not set which is also the default setting, metac binary panics (i.e. metac container stops running) if any of the configured controllers make use of custom resources whose definitions are not yet available in the cluster. The binary panics after exhausting the configured timeout (which defaults to 30 minutes).

This feature help teams to manage the transient phases during controller upgrades. One such scenario is when these upgraded controllers make use of new custom resource definition(s) that are yet to be applied against the cluster.

Signed-off-by: AmitKumarDas amit.das@mayadata.io

AmitKumarDas commented 4 years ago

I will explain the details. I realised I should have explained how it works now vs how it will work after this change

AmitKumarDas commented 4 years ago

@grzesuav done with the rework.

AmitKumarDas commented 4 years ago

And one additional question : what happens if controllers are started and errors happens afterward, like after 20 minutes ? It will restart controller under the hood ?

This is one of the best questions so far. logic will not retry for controllers that fail to discover resources later. For example, if someone deletes a CRD that is used by one of the controllers much later after this controller is in running state, then metac will get logged with errors. It should not panic is what i can remember. However, this is a good enhancement & should be tested.

added an issue to track: https://github.com/AmitKumarDas/metac/issues/144

AmitKumarDas commented 4 years ago

:tada: This PR is included in version 0.4.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: