Retry with `ForcePowerOff` if graceful shutdown times out

[defo89]

Proposed Changes

• I observed that in case Server is stuck in a failed PXE Boot or in other non-responsive state, redfish.GracefulShutdownResetType would not power off the system.
• So we can retry with redfish.PushPowerButtonResetType if above fails.
• Enabled via flag --enforce-power-off

[defo89]

hmm, need to revisit. For Dell it has worked, but on Lenovo I get reset type 'PushPowerButton' is not supported by this service

2024-09-17T12:56:20.219583196Z 2024-09-17T12:56:20Z ERROR   Reconciler error    {"controller": "server", "controllerGroup": "metal.ironcore.dev", "controllerKind": "Server", "Server": {"name":"node006-bb00-system-0"}, "namespace": "", "name": "node006-bb00-system-0", "reconcileID": "1e290486-c86f-4df4-90a5-5c55ab438ae1", "error": "failed to ensure server state transition: failed to ensure server power state: failed to power off server: failed to reset system to power on state: reset type 'PushPowerButton' is not supported by this service"}

[stefanhipfel]

lgtm as well. we could also use metal.ironcore.dev/operation annotation to force a power off. But maybe it is good that this is happening automatically.

[defo89]

lgtm as well. we could also use metal.ironcore.dev/operation annotation to force a power off. But maybe it is good that this is happening automatically.

good point. I'd see metal.ironcore.dev/operation annotation as a troubleshooting tool to perform the operation and also ability to force power off for users that will not use this --enforce-power-off option.