Understanding icenet-driver Makefile

[vbgandhi]

I'm trying to port the icenet driver as a userspace library, and figure out the firemarshal build system. My understanding so far is limited to the following factoids:

1. A symbolic link to the linux submodule at path ~/firemarhsal/boards/default/linux has been created at ~/firemarshal/boards/firechip/linux
2. However, the Makefile in icenet-driver directory sets LINUXSRC=../../../../riscv-linux

At a high-level, if I'm building a custom workload on top of fedora and would like to include a set of headers files, e.g. , #include <linux/interrupt.h> in my custom workloads, how do I compile my workloads? I tried creating a symlink of the linux submodule in the overlay dir of my workload, but it fails to compile and I get the following error -- fatal error: linux/interrupt.h: No such file or directory

[NathanTP]

If you're trying to do this in userspace, it's going to look fairly different than the linux kernel version. Most of the stuff in there is specific to the Linux kernel (like the interrupt.h you mention). The core parts of the driver are really about the low-level API (which commands to which registers do what). What you do with the API is up to you. You'll also need to expose the MMIO registers to userspace from Linux so that the library can control the device. Unfortunately, there isn't great documentation for icenic (we would welcome some).

We also don't have a functional model of icenic right now in Spike which will make it a little trickier to develop. If you end up writing one, I'd love to hear about it. We're currently working on network support in FireMarshal but it's only for QEMU right now.

If you instead meant that you wanted a custom driver that is still going to run in the kernel, that's different. For an example of how to modify the icenic driver for a specific workload, you can look at this unit test: https://github.com/firesim/FireMarshal/tree/master/test/kmods

[vbgandhi]

Thanks, Nathan.

I’m trying to run multi-threaded workloads on fedora for my custom HW configs using firesim but experience significant performance overhead due to syscalls. I’m getting a bandwidth of roughly 0.4 Gbps on a 200Gbps simulation link. Hence, the reason I wish to port the icenet driver to userspace.

I was particularly thinking about the interrupt line between the NIC and CPU, and it seems like to control the NIC entirely in userspace, I’ll have to modify the NIC as well?

High-level question – is there a way to run multi-threaded workloads bare-metal using firesim or any pre-existing work that addresses the syscall issue?

[NathanTP]

Ya, NW performance is pretty poor. As you've deduced, it's mostly due to the Linux NW stack. We spent some time investigating this in detail and improving performance. You can read about it here: https://dl.acm.org/doi/10.1145/3373376.3378455. There is an appendix there with implementation details.

Unfortunately we weren't able to upstream that code for various reasons, and it was focused on in-kernel and HW improvements rather than user-space NW stack which is probably a pretty good route. For that, I'm not 100% sure. There are two main challenges there. First, everything is physically addressed. This means you have to figure out how to pin pages and get their physical addresses from userspace. I know from experience that this isn't simple. Second, you'll need to figure out how to mmap the mmio registers into the user app so it can control the NIC. I don't think it will need HW changes though. We did something similar for the centrifuge project, you can see our code here:

user space utilities: https://github.com/hqjenny/centrifuge/blob/master/workloads/hardcoded/custom_mmap/common.h https://github.com/hqjenny/centrifuge/blob/master/workloads/hardcoded/os_utils.h

custom mmap: https://github.com/firesim/linux/commit/8f5d9162eb07e192dcb4d1562668c685dfa660c5#diff-212a5a1672c9f65330eb8e0bed5ed3cca92f04255f2296cb230ae25f08121e3d

[NathanTP]

Closing: Out-of-scope