Avoid waiting for failed resources

[mortent]

This updates the WaitTask to handle resources that are reported as Failed. It prevents the library from waiting the full length of the wait timeout for resources that we know are unlikely to successfully reconcile.

The behavior is:

• If a resource is reported as Failed, the library will remove it from the set of resources we are waiting to be reconciled, add it to a set of failed resources, and send a WaitEvent with the ReconciledFailed operation.
• Any failed resources will not prevent a WaitTask from being completed if the other resources in the task have become Current. So a resource becoming Failed does not terminate a WaitTask, it just makes sure we don't end up waiting for the timeout to expire when we know resources are unlikely to successfully reconcile.
• If a resource becomes Failed, but happens to become unblocked before all other resources in the task has completed, it can transition out of the Failed state. It can move either into Current or InProgress.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mortent

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/cli-utils/blob/master/OWNERS)~~ [mortent] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[mortent]

@karlkfi Updated the PR with the following changes:

• Updated the WaitEvent to use operation ReconcileFailed rather than ReconcileStalled. Also updated the functions and variable names accordingly.
• Make sure that resources can transition from Failed back to InProgress. Added a test for this.

[mortent]

/retest /cc @ash2k

[ash2k]

I'm sorry, I will not be able to review PRs for some time.

[haiyanmeng]

/lgtm