Building the OS environment and deployment tooling for the data plane project (prototyping)

cesargithedgehog commented 2 months ago

Obs: Any installer/deployment tooling will need to go to fabricator eventually and be done same as for the control node / fabric.

cesargithedgehog commented 2 months ago

References:

daniel-noland commented 2 months ago

Thank you for this. Will review these docs so we can get in line with the other build infra

daniel-noland commented 2 months ago

One point of order to consider:

at some point we will want do build

for a specific CPU (or set of CPUs)
with profile guided optimization (both for the rust parts and the DPDK / rdma-core libraries
with BOLT (likely turn this on at the same time as bolt)

I don't know flatcar / fabricator well enough to tell if this will present a challenge

cesargithedgehog commented 1 month ago

Some Requirements (source: @daniel-noland)

1. modest x86_64 chip in the AMD 4xxx line (basically ryzen 7 or 9 chip but server grade so we get ECC)

2. modest memory 32GB

3. _quite_ fancy network card: ConnectX-7 dual port with crypto offloads

4. modest NVMe drives

We don't need much in the way of hardware outside the NIC because I'm pretty good at DPDK NIC offloading.  Even stateful tables can mostly be offloaded. So the CPU will typically see only the first 1-5 packets of a flow before being hardware offloaded.
This includes EVPN + VXLAN encap / decap, NAT operations on the inner packet, and push/pop of needed VLAN tags.
The software itself is written in rust with automatically generated bindings to DPDK.
The executable we will run is a single, statically compiled object.
It can be the one and only file in a read-only container.

You don't need any distro or anything like that.
Containerd should take care of mounting /sys /proc and /dev as needed in the container.  The exception is that we will need to mount in something like /dev/huge/1G and /dev/huge/2M as hugetlbfs filesystems from outside of the container we run this in.
No requirement for a VM.  It will be significantly easier without a VM in fact

The container needs an unfortunate amount of privileges at the moment.  Basically think docker run --privileged --security-opt seccomp=unconfined ...  for the moment.

I am working on reducing that requirement but it is tricky and it can't be done early in the project.
The container will also need to run with host networking as it needs direct control over physical hardware (i.e. the nic) (edited) 

Finally, we make use of the embedded switch in the ConnectX-7 card.  That will need to be configured before the dataplane app starts up.  I have scripts to configure that switch chip which I can provide.
Currently, that script is in bash and requires tools that are easiest to install in a Debian container.  I would love to rewrite that script in Rust but that requires me to extend the Netlink bindings available in Rust to include devlink as a feature.

This script determines if the kernel has the things we would like to see:


#!/bin/bash

set -euo pipefail

# change this to the path to your config.gz
declare -r CONFIG="/proc/config.gz"

declare -Ar NEEDS=(
  # critical requirements
  [CONFIG_NET_SWITCHDEV]="y" # needs to be y
  [CONFIG_VXLAN]="\(y\|m\)" # needs to be y or m
  [CONFIG_MLX5_CORE]="\(y\|m\)" # needs to be y or m
  [CONFIG_MLX5_INFINIBAND]="\(y\|m\)" # needs to be y or m
  [CONFIG_MLX5_ESWITCH]="y" # needs to be y
  [CONFIG_MLX5_SW_STEERING]="y" # needs to be y

  # eventual requirements
  [CONFIG_MLX5_EN_IPSEC]="y" # eventually needs to be y

  # very strange and frustrating if disabled
  [CONFIG_MLX5_BRIDGE]="y" # should be y
  [CONFIG_MLX5_CLS_ACT]="y" # should be y
  [CONFIG_MLX5_TC_CT]="y" # should be y
)

for need in "${!NEEDS[@]}"; do
  declare match="${NEEDS[${need}]}";
  if ! zgrep "${need}=${match}" "${CONFIG}"; then
    >&2 printf -- "%s\n" "ERROR: ${need} is not set to ${match}"
  fi;
done

cesargithedgehog commented 1 month ago

https://archive.fosdem.org/2022/schedule/event/modding_the_immutable_how_to_extend_flatcar/

cesargithedgehog commented 1 month ago

Guide to building custom Flatcar images from source

https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-modifying-flatcar/

Flatcar Container Linux SDK scripts

The SDK can be used to:

Patch or update applications or libraries included in the Flatcar OS image.
Add or remove applications and/or libraries.
Modify the kernel configuration and add or remove kernel modules included with Flatcar.
Build OS images for a variety of targets (qemu, bare metal, AWS, Azure, VMWare, etc.).
Lastly, the SDK can upgrade SDK packages and build new SDKs.

*https://github.com/flatcar/scripts

cesargithedgehog commented 1 month ago

Take a look at - https://github.com/githedgehog/control-usb-root

githedgehog / dataplane

Building the OS environment and deployment tooling for the data plane project (prototyping) #4