Vrolijk / OSDD

Sharing ideas about using data diodes
Apache License 2.0
34 stars 10 forks source link

CPU isolation for NIC: goal is to reduce/prevent packet loss #10

Open Vrolijk opened 6 months ago

Vrolijk commented 6 months ago

work in progress:

isolate CPU cores find your physical and virtual cores lscpu --all --extended

Isolate the cores (numbers are all virtual cores bound to a physical) sudo gedit /etc/default/grub Find and modify : GRUB_CMDLINE_LINUX="isolcpus=1,3 nohz_full=1,3" save and update Grub: sudo update-grub

Assign IRQ to CPU. NOTE: need to be done after every reboot. Find IRQ for enp1s0 grep enp1s0 /proc/interrupts

Bind IRQ to CPU core echo 2 > /proc/irq/126/smp_affinity more info about binding to CPU https://support.hpe.com/hpesc/public/docDisplay?docId=c01824714&docLocale=en_US

Stop and start the nic:

sudo ip link set enp1s0 down
sudo ip link set enp1s0 up

Monitor NIC: packetloss and memmory usage watch -n0 tc -s qdisc show dev enp1s0 Monitor CPU and interupts > search for enp1s0 watch -n0 cat /proc/interrupts

Validate settings: are the selected cores isolated: cat /sys/devices/system/cpu/isolated is the NIC bound to a specific core cat /proc/irq/126/smp_affinity

research:

Do we need to stop the irqbalance service? stop when needed sudo service irqbalance status sudo service irqbalance stop

research values under cat /sys/class/net/enp1s0/queues/tx-0/