🐛 awsmachine: only register machine to LB when it's running

[r4f4]

What type of PR is this?

/kind bug

What this PR does / why we need it:

The AWS docs [1] state that to register an instance with the Load Balancer target groups, the instance must be running. Currently, CAPA tries to register it while it's still pending, causing the following error:

0314 22:24:18.512176  701419 awsmachine_controller.go:605] "failed to reconcile LB attachment" err=<
    [could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-int-22623': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 17514354-77ac-42b1-a882-489760563bbd, could not register machine to load balancer: could not register control plane instance "i-039bb22b1e8df7c99" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'
        status code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb]
 >
I0314 22:24:18.512325  701419 recorder.go:104] "events: Failed to register control plane instance \"i-039bb22b1e8df7c99\" with load balancer: failed to register instance with target group 'mrb-capa-67-d2pfx-ext-6443': InvalidTarget: The following targets are not in a running state and cannot be registered: 'i-039bb22b1e8df7c99'\n\tstatus code: 400, request id: 84e1849c-abb7-4af9-9220-8791fdc1a3fb" type="Warning" object={"kind":"AWSMachine","namespace":"openshift-cluster-api-guests","name":"mrb-capa-67-d2pfx-bootstrap","uid":"58af1162-380b-4f3e-93fe-c0e81401070e","apiVersion":"infrastructure.cluster.x-k8s.io/v1beta2","resourceVersion":"562"} reason="FailedAttachControlPlaneELB"

Even though this doesn't stop the install from succeeding, let's wait for the instance state to be "running" and with that avoid unnecessary AWS API calls.

[1] https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-register-targets.html#register-instances

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #5037

Special notes for your reviewer:

Checklist:

• [ ] squashed commits
• [ ] includes documentation
• [X] includes emojis
• [ ] adds unit tests
• [ ] adds or updates e2e tests

Release note:

Fix registering instances to Load Balancers when instances are still pending.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[k8s-ci-robot]

Hi @r4f4. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[k8s-ci-robot]

@mtulio: changing LGTM is restricted to collaborators

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5040#pullrequestreview-2148083567): >/lgtm Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[r4f4]

No regressions found in Openshift e2e CI tests: https://github.com/openshift/installer/pull/8678

[richardcase]

/ok-to-test

[k8s-ci-robot]

@r4f4: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-test	d967ea465f11c463590f7cb93c46872c9061a716	link	true	/test pull-cluster-api-provider-aws-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

[r4f4]

/hold I'll look into the failing tests.