Getting Started Guide: Going from (affordable) hardware to production

[danielkauffman]

I've been researching open source software for switch hardware but haven't seen any high level descriptions of how to get from hardware acquisition to production deployment. Hence this guide. However, there are several points on which I'm not clear and could use help.

Perhaps with clarification on the points that are not clear, this thread can be turned into a "Getting Started Guide" for the website?

Here is my understanding of the process:

1. U-Boot is a bootloader for booting ONIE (an installer) or a network operating system (NOS). U-Boot should already be installed on your switch. If U-Boot is installed, skip to the next step. If U-Boot is not installed, but is supported by your switch, you might be able to install U-Boot using a hardware programmer such as a JTAG programmer.
2. ONIE is software for installing a NOS. ONIE should already be installed on your switch. If ONIE is installed, skip to the next step. If ONIE is not installed, but is supported by your switch, use U-Boot to install ONIE.
3. Boot ONIE. Early in the boot process, press any key to access the U-Boot shell, then run onie_rescue to boot ONIE. Alternatively, most NOS include a command for rebooting to ONIE without accessing the U-Boot shell.
4. Use ONIE to install your NOS of choice.

• dentOS and Open Network Linux (ONL). DentOS was recently forked from ONL. The projects seem to be closely related but I haven't been able to find a clear statement describing the relationship between dentOS and ONL. I also haven't been able to find a road map of whether the path forward includes dentOS only, dentOS merged back to ONL, or both dentOS and ONL as distinct projects.
• SONiC often requires proprietary Linux kernel modules.
• Are there any other significant open source NOS?
• Proprietary NOS such as Big Switch OS, Pica OS, Cumulus Linux. I'm not sure who owns who anymore. :-)
• OpenWRT deserves honorable mention here as a NOS which is relatively easy to install and run on a wide variety of consumer and prosumer routers. However, to the best of my knowledge, OpenWRT has limited ability to offload switching to the switch ASIC (for improved performance) and none of the supported hardware features a dedicated management port (for improved security). Note that unlike the other NOS described here, OpenWRT is installed using the per-router process for upgrading firmware, and can not be installed using ONIE.

5. Boot your NOS.
6. Install switching software to manage the switch ASIC.

• dentOS uses switchdev in the Linux kernel to manage the switch ASIC, so does not need additional software to manage switching.
• ONL does not include switching software out of the box. Some switch ASICs advertise support for the OpenNSL, OpenFlow, or Switch Abstraction Interface (SAI) protocols but I've had trouble identifying which hardware can actually be used with which open source switching software.
• I need to do some more research on SONiC.
• Proprietary NOS should already include switching software.

7. Install routing software.

• FRR. This project seems to be gaining momentum.
• BIRD.
• Proprietary NOS should already include routing software.

8. Test and deploy your open stack. :-)

I'm not sure how to identify hardware and open source software that gets you all the way to step 8. I've identified several issues:

• dentOS has a HCL but that list does not include model numbers or architecture. Without model numbers, I haven't been able to identify if any of these are affordably priced.
• ONL has a HCL that states compatibility with the QuantaMesh T1048-LB9 48 port x 1G switch, which is available at a reasonable price point. Using the LB9, I was able to upgrade U-Boot, compile and install ONIE, and compile and install ONL, but I haven't been able to identify switching software for this hardware.
• SONiC has a fairly extensive HCL.
• Is there any open source switching software which supports switch ASICs not just NICs? Are there any lists of which hardware is supported by which software?

Does affordable hardware and software exist for developers and SMB wanting to deploy an open stack? Can hardware plus software be acquired for under $200 per switch? Under $400? Under $800? Under $1200?

The Mellanox SN2000 and SN3000 switches deserve special mention as these are able to run general purpose Linux distros and the switch ASIC is supported by the Linux kernel. Hopefully, other manufacturers will follow suit. Used pricing for these switches currently ranges from $2500-5000 or so depending on the hardware configuration.

[paulmenzel]

Thank you for your write-up. Instead of commercial NOS as opposed to open source NOS, I’d use proprietary NOS.

Are there any other significant open source NOS?

I guess, OpenWRT can be considered a NOS. There is also SONiC, but it often requires proprietary Linux kernel modules.

[danielkauffman]

@paulmenzel Thank you, I've incorporated your comments above.

[danielkauffman]

It looks like SONiC might support the Celestica E1031 which is currently $600-800 used.

[jmpolom]

Another hardware option worth mentioning are the Mellanox Spectrum switches that have fully upstreamed ASIC support. The SN2000 and SN3000 devices do not require any special/proprietary software (besides the firmware blob in linux-firmware) to use. You can install a general purpose Linux distro (like Debian or Fedora) on these devices and the kernel provides everything necessary to fully interact with the switching platform.

The DENT switches are very close to this level of all in kernel support but currently have varying levels of upstreamed platform support (thus why dentOS uses ONLP still), and also varying levels of upstreamed ASIC feature support due to Marvell contributing driver updates directly to dentOS.

[danielkauffman]

@jmpolom Thank you, I've incorporated the SN2000 and SN3000 devices above, and I've reached out to Marvell for clarification on which of their switches support DentOS.

[shivanangi]

@danielkauffman : You can find the Hardware Compatibility List :

https://dent.dev/supportservices/

[hvina]

I am trying to find out if the "Wedge 100BF-32X 32 x 100G QSFP28 switch ports with Tofino 32D." is supported. The ASIC chip set it Tofino . Can the community let me know if this has been tried

[shivanangi]

@hvina: I don't think so.. maybe you need to try SONIC.

[q2dg]

Vyos is another open-source NOS (https://vyos.net/contribute)