Rework running Fence Agent command

[clobrano]

Run the fence agent command asynchronously in a dedicated goroutine on the same controller's container. The goroutine is also responsible to update FAR status with the command outcome. For this reason two new Status Conditions have been added to take into account fence agents failures or timeouts.

The fence agent command has three new, optional, Spec values:

• RetryCount is the number of times the fencing agent will be executed in case of failures (default: 5)
• RetryInterval is the interval between each fencing agent retries (default 5s)
• Timeout is the timeout for each fencing agent execution (default 60s)

---

TODO

• [x] replace fence_ipmilan mock with software mock of exec.Command only
• [x] add fence agent failure case to updateConditions
• [x] use updateConditions in executor
• [x] split change in smaller chunks
• [x] clean up Executor runners Map when goroutine ends
• [x] investigate why e2e tests with AWS cluster time out: flaky tests :shrug:

[openshift-ci[bot]]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clobrano

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/medik8s/fence-agents-remediation/blob/main/OWNERS)~~ [clobrano] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[clobrano]

/test

[openshift-ci[bot]]

@clobrano: The /test command needs one or more targets. The following commands are available to trigger required jobs:

• /test 4.12-ci-bundle-my-bundle
• /test 4.12-images
• /test 4.12-openshift-e2e
• /test 4.12-test
• /test 4.13-ci-bundle-my-bundle
• /test 4.13-images
• /test 4.13-openshift-e2e
• /test 4.13-test
• /test 4.14-ci-bundle-my-bundle
• /test 4.14-images
• /test 4.14-openshift-e2e
• /test 4.14-test
• /test 4.15-ci-bundle-my-bundle
• /test 4.15-images
• /test 4.15-openshift-e2e
• /test 4.15-test

Use /test all to run all jobs.

In response to [this](https://github.com/medik8s/fence-agents-remediation/pull/106#issuecomment-1820849765): >/test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[clobrano]

/test 4.15-openshift-e2e

[clobrano]

Working on a different approach for testing, so you might want to wait before reviewing this PR

[clobrano]

/test 4.14-openshift-e2e

[mshitrit]

Working on a different approach for testing, so you might want to wait before reviewing this PR

Might want to add a [WIP] suffix on the PR until ready for review

[clobrano]

/test 4.14-openshift-e2e

[clobrano]

/test 4.14-openshift-e2e

[clobrano]

/retest

[clobrano]

/test 4.14-openshift-e2e

[clobrano]

/retest

[clobrano]

/retest

[clobrano]

/retest

[razo7]

/lgtm

[clobrano]

/hold waiting for @mshitrit feedback on his change request

[clobrano]

/retest

[clobrano]

/retest

[razo7]

/lgtm

[mshitrit]

I'm unholding this PR since only contain Nits and it has an E2E fix which is relevant for other PRs. As @clobrano mentioned in case needed it can be addressed in followup PR. /unhold