Planning to add NVMe support to OFA cluster

[dledford]

We are preparing to order some NVMe 2.0 spec compliant devices to add to the OFA cluster. They will require a PCI slot because the current machines don't have any motherboard NVMe connectors (to my knowledge, someone can crack one open and check, but I'm pretty sure I'm right). We plan on adding 4 devices, 1 per machine to 4 machines. I've started updating the test_nodes.md doc to make room for NVMe on the table and will work out where we want to make changes. I'm pretty sure one change I'll recommend is that we undo the PCI double slot usage in nodes 9 and 10, which has never worked right anyway, and that frees up a slot in those machines for NVMe devices. Then I'll want to make sure that the 2 remaining machines are opa machines in order to make sure we cover all of the fabric types. We might need to shuffle a few more cards to make sure we get iWARP and NVMe in the same chassis. @JSpewock @lylavoie Please send me the shipping address you want us to use off-line and I'll make sure to ship the items to the right place

[lylavoie]

Current plan:

• Install into Nodes 1 & 2, remove Intel RoCE cards, use PCI1
• Install into Nodes 9 & 10, use PCI3, remove VPI extender

If the server boards do have an NVMe slot, install the drives there, without removing cards.

[lylavoie]

@dledford two of the NVMe drives have been installed into Nodes 9 & 10, as discussed.

I've confirmed both nodes see the storage via their IPMI iLO interfaces. I have not made any changes to the beaker snippets to target SD card for the install still.

[lylavoie]

NVMe drives have been installed in Nodes 1 & 2 as discussed in this issue.