feat: add binary option to ignore api discovery errors

[AmitKumarDas]

ProblemStatement: As a controller developer, I would like to ignore all or specific api discovery related errors during binary startup. This helps in getting the controller get started with the discoveries that were possible.

[grzesuav]

I was thinking about fixing the same in metacontroller repo, but maybe in a slightly better way - don't ignore, but just put some kind of check if al required API resources are registered in discovery - and do not start discovery until it happens . But is is just an idea though, need to see if its implementable

[AmitKumarDas]

Scenario 1

Managing api discovery errors is required when a CRD is being managed (read create) by a controller & the another controller needs to manage corresponding resource (create/update/delete). However since both controllers are started only during metac binary startup phase, it becomes difficult to signal completion of controllers start up activity to metac.

NOTE: This seems to be applicable to using metac as a library & its custom controllers as config files instead of custom resources.

Scenario 2

API discovery errors can also happen if metac binary (read Kubernetes pod) was started before applying a CRD. This is currently managed by letting the binary to recover from discovery error by trying for a max of 30 minutes. In other words, CRD must applied within 30 minutes of starting up metac binary (assuming one or more metac controllers deal with this CRD).

Suggestion

It might be good to retry controllers facing discovery errors in dedicated goroutines. These retries can be in some pre-determined periodic intervals. Metac should also start reconciling resources for those controllers that were successfully started (if these controllers api resources were discovered successfully). This will make each metac controller independent of each other.

In addition, the api discovery feature of metac can be enhanced to retry discovering the specific api resource if it was not found. This is same as fetch from server if cache does not have it. If it was found successfully from the server then add to existing cache. If it was not found at server then return error like existing implementation does.