Provide deadlines or cancelations for all uses of `context.Context`

[devigned]

/kind feature

Describe the solution you'd like An iteration of a reconcile loop should be bounded by a default deadline. In other words, a single execution of reconcile in a controller should not be able to run indefinitely.

As of today, there are several code paths where context.TODO() is used for a context. This context provides no ability to cancel or bound execution on those code paths. Some of these code paths will need to have a context piped in through functions up the call stack, and may have more widespread impact to function signatures.

Examples:

• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/4d682a1017cd6c849c7e73e7646492d0941c4ed5/exp/cloud/scope/machinepool.go#L151-L152
• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/4afe3c2226f885cb1607e7ccf2526bc09b63b816/cloud/scope/machine.go#L214
• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/4778ca551a5ded2b3da694bcea12b4753e8ee163/cloud/scope/cluster.go#L162
• https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/4afe3c2226f885cb1607e7ccf2526bc09b63b816/controllers/azuremachine_controller.go#L75

TODOs:

• [ ] determine a default max deadline for a single reconcile
• [ ] provide a manager cli argument for default max deadline
• [ ] remove usage of context.TODO and replace with bounded contexts

See Also

• https://github.com/kubernetes-sigs/controller-runtime/issues/798
• https://github.com/kubernetes-sigs/controller-runtime/issues/801

[CecileRobertMichon]

/priority important-soon

[devigned]

/assign