search

ironcore-dev / metal-operator

Kubernetes operator for automating bare metal server discovery and provisioning
https://ironcore-dev.github.io/metal-operator/
Apache License 2.0
10 stars 6 forks source link

Fix DELL PowerEdge iDRAC #170

Open damyan opened 4 days ago

damyan commented 4 days ago

The DELL iDRAC needs some time before it can deliver source information. Asking "too soon" leads to HTTP 503: stderr F 2024-11-08T09:24:55Z ERROR Reconciler error {"controller": "server", "controllerGroup": "metal.ironcore.dev", "controllerKind": "Server", "Server": {"name":"mgmt-1-system-0"}, "namespace": "", "name": "mgmt-1-system-0", "reconcileID": "6b4398aa-9256-4ce5-8e29-d23003ed8565", "error": "failed to update server status: failed to get system info for Server: failed to get systems: failed to retrieve some items: [{\"link\":\"/redfish/v1/Systems/System.Embedded.1\",\"error\":\"503: {\\\"error\\\":{\\\"@Message.ExtendedInfo\\\":[{\\\"Message\\\":\\\"iDRAC is currently unable to display any information because data sources are unavailable.\\\",\\\"MessageArgs\\\":[],\\\"MessageArgs@odata.count\\\":0,\\\"MessageId\\\":\\\"IDRAC.2.8.SYS518\\\",\\\"RelatedProperties\\\":[],\\\"RelatedProperties@odata.count\\\":0,\\\"Resolution\\\":\\\"Wait for the data to be available and retry the operation. If the issue persists, contact your service provider.\\\",\\\"Severity\\\":\\\"Informational\\\"},{\\\"Message\\\":\\\"The service is temporarily unavailable. Retry in 30 seconds.\\\",\\\"MessageArgs\\\":[\\\"30\\\"],\\\"MessageArgs@odata.count\\\":1,\\\"MessageId\\\":\\\"Base.1.12.ServiceTemporarilyUnavailable\\\",\\\"RelatedProperties\\\":[],\\\"RelatedProperties@odata.count\\\":0,\\\"Resolution\\\":\\\"Wait for the indicated retry duration and retry the operation.\\\",\\\"Severity\\\":\\\"Critical\\\"}],\\\"code\\\":\\\"Base.1.12.GeneralError\\\",\\\"message\\\":\\\"A general error has occurred. See ExtendedInfo for more information\\\"}}\"}]"}

A retry mechanism should be Implemented to work around the issue.

afritzler commented 4 days ago

@defo89 @stefanhipfel have you seen this on Dell servers before, that the BIOS info after a reboot is not immediately available through Redfish?

defo89 commented 4 days ago

@defo89 @stefanhipfel have you seen this on Dell servers before, that the BIOS info after a reboot is not immediately available through Redfish?

No, I haven't seen this on the Dell servers we have (PowerEdge R660, iDRAC Firmware Version 7.10.30.05)

afritzler commented 4 days ago

We tested it with a PowerEdge R440.

stefanhipfel commented 1 day ago

I added polling to wait for redfish system resources