search

nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
1 stars 0 forks source link

openstack controller ctl-0 fails to boot due to dead onboard battery #598

Closed aabaris closed 1 month ago

aabaris commented 1 month ago

Server: nerc-ctl-0 Serial Number: 8V66CH2 Location: R7-PB-C19 - U30

Issues:

aabaris commented 1 month ago

Request to techsquare:

I would like to request remote help booting a server with lost bios settings and failing remote console access.

I have limited visibility into the device, but was able to tell that failed battery caused server to revert from UEFI back to BIOS, and in turn fail to boot.

I would like to ask for someone to attach a physical crash cart to the server and attempt setting the server back to UEFI, then attempt to boot it.

Server: nerc-ctl-0 (Dell r430) Serial Number: 8V66CH2 Location: R7-PB-C19 - U30

Please verify that the server serial number matches one provided in addition to it's rack location (it should be visible from the front), our Netbox data is still relatively new.

Thank you very much, please let me know if I can provide further information.

imstof commented 1 month ago

@aabaris I was able to get a look at this yesterday evening.

System s/n and location was correct. The moc keys do not have a key for the front of that cabinet, but I had other keys for the pod, and the back of the rack was unlocked so I was able to access vga/usb. I changed the system back to uefi and rebooted. The system hangs at pxe boot for a few minutes and then goes dark. It was not powered down, since I could reboot with ctl-alt-del, but it was otherwise black screen and unresponsive (monitor is flapping no-signal). I allowed it to try to boot from disk and got to grub menu, but then the system went dark again without any output from os boot.

re: idrac reset. There is only an option to reset to factory default, I couldn't find a reboot option in bios like one has with racadm or idrac gui. A full a/c drain would accomplish the same thing but I didn't want to drain power without access to the front of the box to power on again

Let us know what you want to do next here. Also if you have any idea what key-set has the key for the cabinet doors.

-Christophe

aabaris commented 1 month ago

Thank you, Christophe!

It looks like flip to UEFI was successful, I found the system booted to the OS image that we need. This addresses our most immediate issue and we will pursue further repairs as a separate issue (either ourselves or via another request).

Regarding key access, it's unlocked using harvard's keys. I will make a note for myself to include this information in future requests.

Thank you again for your help. We can consider this request take care of!