Install/setup Intel NICs for KubeCon CNF testing in Packet.net m2.xlarge.x86 systems

lixuna commented 6 years ago

Goal: Acquire 9 NICS from Intel for KubeCon CNF testing on Packet.net

Tasks:

[x] start an new email thread to request 9 NICS from Intel
[x] send shipping info
[x] send model of card needs
[x] include Dell System type
[x] Send email to:
"Cahill, Francis"
"Johnson, Brian"
"Ramakrishnan, Kuralamudhan"
"Yoshinaka, Brian"
Lucina Stricko
Taylor Carpenter
Ed Warnicke
"Maciek Konstantynowicz (mkonstan)"
Zachary Smith
Michael Hsieh
Sam Machiz
Dan Kohn
King, Philip C

--

pcie NICs

[x] NICs are sent from Intel to Packet
[x] Packet validates that NICs are as expected
- Did not work with server

dual port NICs

[x] Packet receives alternative NICs
- enough for 5 of 9 servers
[x] Packet validates that NICs are as expected
[x] Packet installs alternative NICs to servers
[x] Verification by CNF comparison team
- [x] 4 ports show in host OS
- SKIPPED - PCI slot and NUMA validation

Quad port

[x] Packet receives desired quad port NICs
[x] Packet installs quad port NICs to servers
[x] Packet verifies that NICs are working as expected
[x] Verification by CNF comparison team
- [x] 4 ports show in host OS
- [ ] - PCI slot and NUMA validation

taylor commented 6 years ago

Packet.net system information for NF test machines

Details

System base model: Dell R640
CPU: Dual Intel Xeon Gold 5120 2.2Ghz
Memory: 384 GB of DDR4 ECC RAM
Storage: 2 × 120 GB SSD, 3.2 TB of NVMe Flash
Network capability: 10Gbps w/ LACP

Based on Packet.net machine type: m2.xlarge.x86 - https://www.packet.com/cloud/servers/m2-xlarge/

This is based on Dell R640 - Intel Scalable - https://i.dell.com/sites/csdocuments/Shared-Content_data-Sheets_Documents/en/us/PowerEdge-R640-Technical-Guide.pdf

taylor commented 6 years ago

NIC needs and information

quad port card
10Gbps per port
half-height or daughter card (rNDC)

First choice: https://www.dell.com/en-us/shop/dell-intel-x710-quad-port-10gb-da-sfp-network-daughter-card/apd/555-bckl/networking

An alternative is a PCIe card. Ideally, we would need something like: https://ark.intel.com/products/83965/Intel-Ethernet-Converged-Network-Adapter-X710-DA4

The unit would need to be half height to fit in the Dell system

lixuna commented 6 years ago

10-24-2018 update: The wrong NICs were sent. Another set is being dispatched.

taylor commented 6 years ago

There was a switch to 2 dual port Intel X710 NICs because of issues putting the single quad port NIC into the Packet.net systems.

As of today 5 systems have been setup with 2 x 2 port cards

NICs for the remaining 4 systems have been ordered.

Summary of PCI slot locations for 2 dual port Intel X710 NICs:

Detailed PCI device information:

@mackonstan / @edwarnicke / @pmikus / @Snergster, please review.

taylor commented 6 years ago

Intel Quad port x710 NICs were swapped into the systems.

The daughter card (rNDC) versions of the quad port NICs arrived and were installed in the system rather than using the dual port NICs.

https://www.dell.com/en-us/shop/dell-intel-x710-quad-port-10gb-da-sfp-network-daughter-card/apd/555-bckl/networking

OS tested: Ubuntu 18.04 LTS

lspci output:

19:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
19:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)

Detailed version:

19:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
    Subsystem: Dell Ethernet 10G 4P X710 SFP+ rNDC
    Flags: bus master, fast devsel, latency 0, IRQ 44, NUMA node 0
    Memory at a1000000 (64-bit, prefetchable) [size=16M]
    Memory at a2018000 (64-bit, prefetchable) [size=32K]
    Expansion ROM at a2400000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number d0-b1-2e-ff-ff-4b-43-e4
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1b0] Access Control Services
    Capabilities: [1d0] #19
    Kernel driver in use: i40e
    Kernel modules: i40e

19:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
    Subsystem: Dell Ethernet 10G X710 rNDC
    Flags: bus master, fast devsel, latency 0, IRQ 44, NUMA node 0
    Memory at a0000000 (64-bit, prefetchable) [size=16M]
    Memory at a2010000 (64-bit, prefetchable) [size=32K]
    Expansion ROM at a2480000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number d0-b1-2e-ff-ff-4b-43-e4
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: i40e
    Kernel modules: i40e

19:00.2 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
    Subsystem: Dell Ethernet 10G X710 rNDC
    Flags: bus master, fast devsel, latency 0, IRQ 44, NUMA node 0
    Memory at 9f000000 (64-bit, prefetchable) [size=16M]
    Memory at a2008000 (64-bit, prefetchable) [size=32K]
    Expansion ROM at a2500000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number d0-b1-2e-ff-ff-4b-43-e4
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: i40e
    Kernel modules: i40e

19:00.3 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
    Subsystem: Dell Ethernet 10G X710 rNDC
    Flags: bus master, fast devsel, latency 0, IRQ 44, NUMA node 0
    Memory at 9e000000 (64-bit, prefetchable) [size=16M]
    Memory at a2000000 (64-bit, prefetchable) [size=32K]
    Expansion ROM at a2580000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
    Capabilities: [a0] Express Endpoint, MSI 00
    Capabilities: [e0] Vital Product Data
    Capabilities: [100] Advanced Error Reporting
    Capabilities: [140] Device Serial Number d0-b1-2e-ff-ff-4b-43-e4
    Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
    Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
    Capabilities: [1a0] Transaction Processing Hints
    Capabilities: [1b0] Access Control Services
    Kernel driver in use: i40e
    Kernel modules: i40e

ip a output:

2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether e4:43:4b:2e:b1:d0 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:2e:b1:d1 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:2e:b1:d2 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:2e:b1:d3 brd ff:ff:ff:ff:ff:ff

taylor commented 6 years ago

@pmikus, @mackonstan, @michaelspedersen

Please review the pci slots used w/regards to NUMA and other considerations and add comments here. Specifically if there are any issues or if the configuration is sufficient

cncf / cnf-testbed