support drivers needed on packet.net

justincormack commented 7 years ago

Would be nice to have some support on packet.net as it can run VMs.

justincormack commented 7 years ago

Needs BYO OS support see https://trello.com/c/9FTwpsNM/19-byo-os-support-via-custom-ipxe

avsm commented 7 years ago

https://twitter.com/packethost/status/837665802404900864

Should be ready next week they say

avsm commented 7 years ago

I have iPXE working on packet.net now, and booting via netboot.xyz. Creating a custom Moby target now...

justincormack commented 7 years ago

yay! I guess we need some more kernel options for the hardware? what does lspci show?

avsm commented 7 years ago

Took a couple of iterations but Ubuntu boots on it now; lspci looks like this (on a Type 0)

$ lspci
00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 2.5 GbE Backplane (rev 03)
00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 2.5 GbE Backplane (rev 03)
00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02)
00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02)
00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
02:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
03:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

rn commented 7 years ago

the network cards needs the IGB driver added. Not sure if the AHCI/SATA requires a special driver

justincormack commented 7 years ago

I think we have AHCI. We might also have IGB I forget.

Ah thats the 2.5Gb backplane part.

Also worth booting one of the big machines, may as well at the 10Gb support too (again I added some drivers for 10Gb, at least the Intel VF part, maybe more)

avsm commented 7 years ago

Notes from packetcommunity Slack discussion:

Can use DHCP but they recommend querying a metadata URL and static IP, but DHCP will work for the lifetime of a machine provision.
Packet.net needs serial on ttyS1 at 115200.
The serial SOS (over ssh) is hooked up to this, so the kernel command-line also needs something like ro serial console=tty0 console=ttyS1,115200n8
ARM64 is a bit different than x86_64, they recommend provisioning one and checking the getty setup.
There is a bug in their preprovisioned Ubuntu images that don't setup serial console correctly due to the switch to systemd. Use CentOS7 image instead as one that works.
iPXE URL booting works fine, just need to customise it with the above and then can look at the kernel booting serial output for ensuring the right kernel drivers exist for all the different hardware.

avsm commented 7 years ago

This iPXE is sufficient to boot on a Type 0 machine:

#!ipxe

dhcp
set base-url http://recoil.org/~avsm/ipxe
set kernel-params ip=dhcp nomodeset ro serial console=ttyS1,115200
kernel ${base-url}/moby-bzImage ${kernel-params}
initrd ${base-url}/moby-initrd.img
boot

avsm commented 7 years ago

Type 1 also boots successfully to console.

avsm commented 7 years ago

The Type 2 looks like a Mellanox NIC:

00:00.0 Host bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 (rev 01)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 (rev 01)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 (rev 01)
00:05.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management (rev 01)
00:05.1 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug (rev 01)
00:05.2 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors (rev 01)
00:05.4 PIC: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC (rev 01)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02)
02:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
05:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
06:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
07:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe-PCI Bridge [PPB]
08:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)
7f:08.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
7f:08.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
7f:08.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 0 (rev 01)
7f:09.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
7f:09.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
7f:09.3 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D QPI Link 1 (rev 01)
7f:0b.0 System peripheral: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
7f:0b.1 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)
7f:0b.2 Performance counters: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 (rev 01)

rn commented 7 years ago

can you check which module was loaded for it? should be mlx something. ethtool has an option to tell you the driver. The MLX stuff may require multiple modules, IIRC.

Also the RAID controller likely needs a driver too

avsm commented 7 years ago

# ethtool -i p2p1
driver: mlx4_en
version: 2.2-1 (Feb 2014)
firmware-version: 2.30.8040
expansion-rom-version: 
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
# lsmod|grep mlx
mlx4_en               110592  0
vxlan                  49152  2 ixgbe,mlx4_en
ptp                    20480  2 ixgbe,mlx4_en
mlx4_core             286720  1 mlx4_en

I think CONFIG_NET_VENDOR_MELLANOX might be enough for this; trying that now.

ijc commented 7 years ago

That'll just make more options available, which you'll want to enable I think, see drivers/net/ethernet/mellanox/mlx*/Kconfig for those.

rn commented 7 years ago

so needs mlx4_core and mlx4_en. The latter is the Ethernet driver. IIRC, there is also a Inifiniband driver.

config MLX4_EN
        tristate "Mellanox Technologies 1/10/40Gbit Ethernet support"
        depends on MAY_USE_DEVLINK
        depends on PCI
        select MLX4_CORE
        select PTP_1588_CLOCK
        ---help---
          This driver supports Mellanox Technologies ConnectX Ethernet
          devices.

config MLX4_EN_DCB
        bool "Data Center Bridging (DCB) Support"
        default y
        depends on MLX4_EN && DCB
        ---help---
          Say Y here if you want to use Data Center Bridging (DCB) in the
          driver.
          If set to N, will not be able to configure QoS and ratelimit attributes.
          This flag is depended on the kernel's DCB support.

          If unsure, set to Y

config MLX4_CORE
        tristate
        depends on PCI
        default n

config MLX4_DEBUG
        bool "Verbose debugging output" if (MLX4_CORE && EXPERT)
        depends on MLX4_CORE
        default y
        ---help---
          This option causes debugging code to be compiled into the
          mlx4_core driver.  The output can be turned on via the
          debug_level module parameter (which can also be set after
          the driver is loaded through sysfs).

Looks like MLX4_EN needs to be set otherwise it builds it as a module

justincormack commented 7 years ago

I think it should come up ok with dhcp as I think the default setting is dhcp all interfaces and I dont think we hardcode eth0 anywhere any more hopefully. I dont actually know if it will be called p2p1, Linux net device naming got all mysterious.

justincormack commented 7 years ago

We could build these as modules rather than builtin...

ijc commented 7 years ago

It's udev/biosdevname/network-manager which cock up the names I think, the kernel still defaults to eth<N>, unless you've taken deliberate steps they ought to come out sane...

justincormack commented 7 years ago

It makes sense for the VF devices, so you can keep track of their parentage, but the names are quite confusing.

avsm commented 7 years ago

CONFIG_NET_VENDOR_MELLANOX was enough to detect the eth0/eth1 in the Type2 machines, but still need to do the dance to rename and setup the bond0 interface before I can send traffic.

justincormack commented 7 years ago

You need bonding for it to work? I guess that should be a tiny container that bonds, as it is not a usual default.

avsm commented 7 years ago

Yeah looks like they use LACP mode 4 on the higher end Type 2 and Type 3 to split up their networks: https://www.packet.net/help/kb/how-does-the-packet-bonded-network-interface-work/

justincormack commented 7 years ago

ok, I think a tiny container that just sets bonding called packet-bond or something would be the correct solution.

rn commented 7 years ago

I'm inclined to close this as most of the driver support is in and then open a separate issue for the bonding container?

ghost commented 7 years ago

The kernel also needs CONFIG_MEGARAID_SAS=m for the RAID controller.

linuxkit / linuxkit

support drivers needed on packet.net #1245