Detach annotation does not apply immediately

[lentzi90]

What steps did you take and what happened:

In the e2e test where we test the detach annotation, we had to add a sleep before attempting to delete the BareMetalHost. Otherwise it would still go into deprovisioning.

What did you expect to happen:

As soon as the detach annotation is set, the BareMetalHost should be possible to delete without causing any change to its state.

Anything else you would like to add:

Here is what it looks like when the test fails because the BMH goes to deprovisioning:

[2024-05-17T03:11:49.176Z]   STEP: Waiting for the BMH to become provisioned @ 05/17/24 03:08:16.862
[2024-05-17T03:11:49.176Z]   STEP: Verifying the node booting from disk @ 05/17/24 03:10:57.201
[2024-05-17T03:11:49.176Z]   INFO: System is booted from a disk.
[2024-05-17T03:11:49.176Z]   STEP: Retrieving the latest BMH object @ 05/17/24 03:11:12.581
[2024-05-17T03:11:49.176Z]   STEP: Adding the detached annotation @ 05/17/24 03:11:12.619
[2024-05-17T03:11:49.176Z]   STEP: Saving the status to a JSON string @ 05/17/24 03:11:12.969
[2024-05-17T03:11:49.176Z]   STEP: Deleting the BMH @ 05/17/24 03:11:12.97
[2024-05-17T03:11:49.176Z]   STEP: Waiting for the BMH to be deleted @ 05/17/24 03:11:15.243
[2024-05-17T03:11:49.176Z]   [FAILED] in [It] - /home/metal3ci/workspace/metal3-bmo-e2e-test-periodic/test/e2e/common.go:101 @ 05/17/24 03:11:16.017
[2024-05-17T03:11:49.176Z]   INFO: Deleting namespace provisioning-ops-5sx97e
[2024-05-17T03:11:49.176Z]   << Timeline
[2024-05-17T03:11:49.176Z] 
[2024-05-17T03:11:49.176Z]   [FAILED] Told to stop trying after 0.771s.
[2024-05-17T03:11:49.176Z]   BMH provisioning-ops in namespace provisioning-ops-5sx97e should be deleted
[2024-05-17T03:11:49.176Z]   BMH is in an unexpected state: deprovisioning
[2024-05-17T03:11:49.176Z]   At one point, however, the function did return successfully.
[2024-05-17T03:11:49.176Z]   Yet, Eventually failed because the matcher was not satisfied:
[2024-05-17T03:11:49.176Z]   Expected
[2024-05-17T03:11:49.176Z]       <bool>: false
[2024-05-17T03:11:49.176Z]   to be true
[2024-05-17T03:11:49.176Z]   In [It] at: /home/metal3ci/workspace/metal3-bmo-e2e-test-periodic/test/e2e/common.go:101 @ 05/17/24 03:11:16.017
[2024-05-17T03:11:49.176Z] 
[2024-05-17T03:11:49.176Z]   Full Stack Trace
[2024-05-17T03:11:49.176Z]     github.com/metal3-io/baremetal-operator/test/e2e.WaitForBmhDeleted({0x38a60e8?, 0xc0002387d0}, {{0x38b8980, 0xc000d36240}, {0xc001b62df0, 0x10}, {0xc00005b998, 0x17}, {0xc000d25590, 0x3, ...}}, ...)
[2024-05-17T03:11:49.176Z]      /home/metal3ci/workspace/metal3-bmo-e2e-test-periodic/test/e2e/common.go:101 +0x23f
[2024-05-17T03:11:49.176Z]     github.com/metal3-io/baremetal-operator/test/e2e.glob..func9.2()
[2024-05-17T03:11:49.176Z]      /home/metal3ci/workspace/metal3-bmo-e2e-test-periodic/test/e2e/provisioning_and_annotation_test.go:163 +0x1476

Environment:

• Baremetal Operator version: main
• Environment (metal3-dev-env or other): BMO e2e

/kind bug

[lentzi90]

/triage accepted

[kashifest]

/help

[metal3-io-bot]

@kashifest: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/metal3-io/baremetal-operator/issues/1729): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[metal3-io-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[dtantsur]

Would be curious to see BMO logs, at least the relevant part.

[Rozzii]

/remove-lifecycle stale

[metal3-io-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues will close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[lentzi90]

/remove-lifecycle stale

[Rozzii]

/lifecycle frozen