figure out why ipxe boot doesn't work with broadcom qlogic nic in the FC430 in the new research rack

[naved001]

I have been already working on it but opening an issue now.

This is the NIC we have:

*-network:0
  description: Ethernet interface
  product: NetXtreme II BCM57810 10 Gigabit Ethernet
  vendor: Broadcom Inc. and subsidiaries
  version: 10
  size: 10Gbit/s
  capacity: 10Gbit/s
configuration: autonegotiation=on broadcast=yes driver=bnx2x driverversion=5.15.0-83-generic duplex=full firmware=FFV16.10.04 bc 7.14.18

The device and vendor ID is 14E4 168E:

root@oct10-191:~# lspci |grep -i net
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
root@oct10-191:~# grep PCI_ID /sys/bus/pci/devices/0000\:01\:00.0/uevent
PCI_ID=14E4:168E
root@oct10-191:~# grep PCI_ID /sys/bus/pci/devices/0000\:01\:00.1/uevent
PCI_ID=14E4:168E

The problem is that Ironic cannot boot this system into inspection mode because the NIC is not available in ipxe. According to https://ipxe.org/appnote/hardware_drivers this device ~is simply not supported.~ does not have a native ipxe driver.

I confirmed that the NIC is not available in ipxe by going into the ipxe shell instead of starting the boot process

DEFAULT ipxe
LABEL ipxe
  KERNEL ipxe.lkrn
  APPEND shell

and then running ifstat which shows us no interfaces are available.

But where this gets weird is that, at the boot process we get an option to hit CTRL+B to get an ipxe shell, and that somehow has access to the NIC and we can successfully boot from it and inspection passes (I manually run the chain https://our-controller/boot.ipxe to finish the boot process)

But if I get the shell in "our" version of ipxe I don't see the NIC at all:

Notice that the version of ipxe is different (1.0.0+ vs 1.20.1+). I don't know where this different ipxe is coming from. I have looked at the NIC configuration and it only has pxe on it, not ipxe. If we disable the switchports and no pxe boot is happening then there's no option to hit CTRL+B to get this working ipxe shell.

I also tried to compile ipxe with the device ID of our NIC added but that didn't work. It causes a non-maskable interrupt (NMI) when ipxe initializes the devices.

Apparently, the OCT4 rack has some of the same NICs and those seem to boot fine. I verified from idrac that it is the same Broadcom NIC, though I have not verified if it's the exact same PCI device ID and firmware.

[naved001]

@hakasapl @larsks I have added more details to the description.

@tzumainn it would be nice to get hands on some of those OCT4-cha* systems where this works to verify the device id and firmware version.

[tzumainn]

@naved001 would it be sufficient to ask someone on the OKD side to run the relevant commands to get us the information?

[naved001]

@tzumainn that should be fine I guess.

Btw, I am reading some forum threads https://forum.ipxe.org/showthread.php?tid=6989&pid=9465#pid9465

You usually use a .pxe when you want to use native iPXE drivers, most likely loading ipxe.pxe using chainloading.

And .kpxe is most commonly used with undionly.kpxe, because it allows you to use the driver of the vendor PXE ROM (which is handy when iPXE doesn't support your card, or the iPXE driver is broken in some way).

So, I suspect that we somehow boot undionly.kpxe first which gives us that ipxe with the working nic.

[naved001]

Apparently, the OCT4 rack has some of the same NICs and those seem to boot fine. I verified from idrac that it is the same Broadcom NIC, though I have not verified if it's the exact same PCI device ID and firmware.

It is the same PCI device ID, but it's running a slightly older firmware version.

*-network:0
description: Ethernet interface
product: NetXtreme II BCM57810 10 Gigabit Ethernet
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:01:00.0
logical name: eth0
version: 10
serial: a8:99:69:a7:0a:01
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list rom ethernet physical fibre autonegotiation
configuration: autonegotiation=on broadcast=yes driver=bnx2x driverversion=1.713.36-0 storm 7.13.15.0 duplex=full firmware=FFV15.00.14 bc 7.14.12 ip=192.168.55.79 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
resources: irq:39 memory:93000000-937fffff memory:93800000-93ffffff memory:94010000-9401ffff memory:91a00000-91a7ffff
*-network:1
description: Ethernet interface
product: NetXtreme II BCM57810 10 Gigabit Ethernet
vendor: Broadcom Inc. and subsidiaries
physical id: 0.1
bus info: pci@0000:01:00.1
logical name: eth1
version: 10
serial: a8:99:69:a7:0a:04
width: 64 bits
clock: 33MHz
capabilities: bus_master cap_list rom ethernet physical fibre autonegotiation
configuration: autonegotiation=on broadcast=yes driver=bnx2x driverversion=1.713.36-0 storm 7.13.15.0 firmware=FFV15.00.14 bc 7.14.12 latency=0 link=no multicast=yes port=fibre
resources: irq:51 memory:92000000-927fffff memory:92800000-92ffffff memory:94000000-9400ffff memory:91a80000-91afffff

[naved001]

it looks like downgrading to firmware V15.00.14 causes inspection to pass. So, at this point we know what the fix is but we don't know the root cause.

This NIC doesn't have a native ipxe driver, so the only way this works is if ipxe is using the UNDI driver. I will read the changelog to see what changed between version 15 and 16 of this NIC's firmware.

[naved001]

I got the firmware from here: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=68cj1&oscode=w12r2&productcode=poweredge-r730&dgc=SM&cid=376139&lid=spr3676281897&linkId=99352685

[naved001]

I tested firmware version 15.20.13 as that was the most recent version on the original link and that also works.

While that page doesn't link to version 16.10.04 (which the problem machines nic's were running), I got it from here: https://www.dell.com/support/home/en-in/drivers/driversdetails?driverid=d4n9t and I forced reinstalled the same version on a problem host and then that host was fixed.

[centos@nice-chow ~]$ sudo !!
sudo ./Network_Firmware_D4N9T_LN64_16.10.04_A00-00_01.BIN
Collecting inventory...
..
Running validation...

BCM57810 10 Gigabit Ethernet rev 10 (eno1)

The version of this Update Package is the same as the currently installed version.
Software application name: BCM57810 10 Gigabit Ethernet rev 10 (eno1)
Package version: 16.10.04
Installed version: 16.10.04

BCM57810 10 Gigabit Ethernet rev 10 (eno2)

The version of this Update Package is the same as the currently installed version.
Software application name: BCM57810 10 Gigabit Ethernet rev 10 (eno2)
Package version: 16.10.04
Installed version: 16.10.04

Continue? Y/N:y
Y entered; update was forced by user
Executing update...
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER PRODUCTS WHILE UPDATE IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!

It looks like there's nothing wrong with the firmware version itself.

@hakasapl is testing deploying this version using Dell OME (which was originally used to update all the FC430s) to see if that somehow breaks things.

[hakasapl]

Node doesn't boot with the OME version of the firmware, despite the version reported matching with the one on the website.

[tzumainn]

Would it make sense to test having OME install an older version of the firmware that Naved has also verified works? If that fails as well, then maybe the problem has something to do with OME itself.

[joachimweyl]

@naved001 what are the next steps on this issue?

[naved001]

@joachimweyl We could try out what Mainn said i.e. install an older firmware using OME and see if that also fails.

Overall, this is now low on my priority list. We kind of know it's something to do with the firmware that OME installed, and we also have a solution. It depends on how much more digging we want to do. Additionally, we have decided to not even power on the FC430s that are affected by this issue.

We should probably manually install the NIC firmware for FC430s at some point, but because these won't be in service it can wait.

[hakasapl]

@naved001 given the scope of this issue is to find the issue, I think we can close this?

[naved001]

@hakasapl Sure