H700 bricked (Dell T710)

H700 in Dell PowerEdge T710

I moved thus placing my T710 (Dell series 6, Dell 11th generation, circa 2009) in long-term storage.

Came the time to power it back up, the fan roared, and I get no LCD panel, no beep, no Ethernet NIC flickering, no nothing.

Hooked up my last remaining old SVGA monitor and I am greeted with this:

Invalid PCIe card found in the Internal Storage slot
System halted.

Ctrl-R (PE Expandable RAID Controller BIOS), Ctrl-E, Ctrl-I, Ctrl-S (PXE config), F1 (help), F2, F5, F6 (H700 firmware, F11 (switch-between-controllers), nor F12 (boot-by-controller/PXE-boot) keystroke sequences cannot get past this CPU-halted stage. It is almost equivalent to being bricked.

Dell PowerEdge T710 Hardware Owner’s Manual has only this to say about the above error message but it wasn’t an helpful message and rather cryptic.

The system halted because an invalid PCIe expansion card is installed in the dedicated storage controller slot.

Remove the PCIe expansion card and install the integrated storage controller in the dedicated slot.

The Internal Storage slot is a bracket-less 8-bit PCIe 2.0 expansion slot and is currently populated with a Dell H700 Integrated RAID SAS/HDD/SDD controller adapter having 2x4 SAS connector (which is diametrically different than H700 modular having 1x4 SAS) and a BBU write-back cache.

There is an external battery hooked up to the controller. My trusty voltmeter says battery is near-dead. Ordered and installed a replacement battery. This is not always necessary as it is still rechargeable after long-term storage bleed off but it increases your reliability after 7- month off-grid storage.

Fast forward 5 days, installed replacement battery and the VGA monitor still says:

Invalid PCIe card found in the Internal Storage slot

Argh!

Reading up on many forums, I gathered that the H700 (as well as H200) controller can have its PCID be reprogrammed into one of several different types of controllers.

Who knew? A 5-in-1 PCI card depending on its programmable PCI PCID value. That PCID value in NVRAM got resetted after rechargeable battery exhausted due a long term being off-grid.

Now comes the hard parts:

1.  How to do this NVRAM reprogramming, and 
2.  what PCID value to choose.

Given the fact that T710 BIOS was halted and made into a near-bricked state, it is impossible to do the NVRAM reprogramming of this controller adapter within the same T710 PC (because BIOS wont boot into a shell thus not allowing me to use a flash utility to fix itself).

Wrote A and B separately on each of the two SFF-8482 x4 drive connector cables that are connected to this SATA/ SAS PCI controller and disconnected them. Useful when putting it back where it belongs (it will not work if replaced backwardly).

Removed SATA/SAS PCI controller card from T710 and inserted into another (powered-down) PC having been installed and running with Linux.

Boot up second PC. Log into shell. Cloned the lsi tool from URL.

Downloaded Dell firmware from Dell website: DELL PERC H700 Integrated firmware release 12.10.7-0001

Troubleshootings mmbus

For anyone in the future, if you get a Linux error about "mmap bar1: Invalid argument" this is what you do to add iomem=relaxed to the Linux kernel boot cmdline:

Step 1: Edit /etc/default/grub Step 2: Find the line with GRUB_CMDLINE_LINUX_DEFAULT and add iomem=relaxed inside the quotes Step 3: Run update-grub Step 4: Reboot

Source: https://github.com/marcan/lsirec/issues/6

H700 PCID

Similarly to what you've shown for the H200, the H700 has PCI PCID codes for "internal" and "Adapter". T710 uses this card as an “Internal” PCI which is not seated on motherboard directly but side-installed on a special slot and call that card configuration option as an “Integrated”.

grep H700 /usr/share/hwdata/pci.ids   

1028 1f16  PERC H700 Adapter (T710 semi-bricked)
1028 1f17  PERC H700 Integrated (T710 needs this)
1028 1f18  PERC H700 Modular
1028 1f1b  PERC H700 Integrated

So, the native "Integrated" card shows:

lspci -Dmmnn | grep LSI
0000:01:00.0 "RAID bus controller [0104]" "LSI Logic / Symbios Logic [1000]" "MegaRAID SAS 2108 [Liberator] [0079]" -r05 "Dell [1028]" "PERC H700 Integrated [1f17]"

Change PCID from LSI to Dell

Had to change the SubsysVID from LSI to DELL (1028).

Hardware Owner’s Manual

Sources:

•   https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=9fvj2&oscode=ws8r2&productcode=poweredge-t710
•   https://forums.servethehome.com/index.php?threads/guide-flashing-h310-h710-h810-mini-full-size-to-it-mode.27459/page-13
•   https://www.reddit.com/r/homelab/comments/4gb0kg/h200_in_storage_slot_is_a_no_go/
•   https://serverfault.com/questions/655368/swapping-perc-6-i-with-perc-6-ir-in-dell-r710-invalid-pci-card-error
•   https://www.dell.com/support/kbdoc/en-us/000131648/list-of-poweredge-raid-controller-perc-types-for-dell-emc-systems
•   https://hardforum.com/threads/need-help-with-h200-freenas-and-dell-r710.1891289/
•   https://www.dell.com/community/PowerEdge-HDD-SCSI-RAID/Re-flash-PERC-H700-Firmware-from-Adapter-to-Integrated-Adapter/td-p/4440877/page/4
•   https://forums.overclockers.com.au/threads/a-cheaper-m1015-the-dell-h200-and-a-howto-flash.1045376/page-3
•   https://web.archive.org/web/20160204070852/https://rghost.net/8NCrbXMN5
•   https://github.com/marcan/lsirec
•   https://phoxden.net/H310MM_IT.pdf 
•   https://web.archive.org/web/20201124175634/https://static.phoxden.net/H310MM_IT.pdf
•   https://www.youtube.com/watch?v=v0AEHVdc_go
•

Versions ⁃ BIOS v3.18.0 ⁃ PERC H700 package v12.10.7-001 ⁃ PERC H700 Integrated BIOS Configuration Utility/Firmware v2.02-0025.1 ⁃ ST2000NM0023 vGE11 ⁃ ST3300657SS v0006 ⁃ IBM-ESXS ST32000444SS vBC28

eLvErDe / hwraid

H700 bricked (Dell T710) #141