Working around loss of kernel stack when ZC is active

[mzpqnxow]

Hello,

This may be more appropriate for a mailing list or other forum, if so please let me know

I have a few servers that only have a single physical network link. For these, while my zc application runs, I lose the ability to manage it via SSH, as expected

Unfortunately, I do NOT have any options in terms of adding another network link for these specific systems. Normally I just add a cheap 100Mbps management link, but I'm constrained here

So, I'm wondering if there are any good workarounds for the fact that, once in ZC mode, all networking is lost to/from the system, because the kernel stack no longer handles the traffic. I would like to be able to use ssh while the application is running in ZC mode

For context, my zc application does high speed transmit, lots of small TCP packets, <100 bytes each. I use only a single source port, to reserve the right to steer on either side if ever needed. The receive volume is only ~5% of that amount.

Are there any clever options for maintaining the ability to manage the system via SSH while my application is running, in a way that doesn't completely negate the performance that zc is offering?

My ideas thus far:

• Current "solution": cron interrupts my application for 5 minutes every 12 hours, at which point the system can be accessed via SSH for maintenance, if necessary. Low-tech, reliable solution. Doesn't allow me to debug or monitor the application or system while my application is running, though
• I noticed the zbounce tool seems to be able to receive on a zc queue and inject what it receives into the kernel stack. Could I run this concurrently with my zc application, and just have it inject tcp/22 traffic into the kernel stack? Would that be sufficient to allow SSH to work?
• If my NIC supports VFs, is there a way to have a VF that is in ZC mode and a VF used by the kernel stack? I'm aware of VFs but have no experience using them. The documentation I've read is primarily focused on using VFs to facilitate virtualization. I suspect the answer to this may even depend upon the NIC/driver
• Is it technically possible to implement a one-way/tx-only ZC mode using the pfring and zc infrastructure? In my case, I want the ability to bypass the kernel for tx only for my application. I don't care as much about performance in the received packets, I would be perfectly happy using a standard raw socket via libpcap for receive if it meant I could keep my kernel tcp/ip stack
• If zbounce isn't the right tool for the job, perhaps an actual userland TCP/IP stack (made ZC aware) could stand in as the network stack for just tcp/22.. I don't see why not, but it may be a bit of work finding and (probably) patching one to use ZC

Any insight or guidance is appreciated. I've read a LOT of documentation over the past couple of years, but it's proven very difficult to conclude anything. I know I'm an edge case, as usual... 😔

Thanks

[cardigliano]

@mzpqnxow the only way to support this, is to handle host traffic forwarding it to the stack from the application which is running ZC (nScrub for instance is doing this), but this required non-trivial changes to the application.

[mzpqnxow]

@mzpqnxow the only way to support this, is to handle host traffic forwarding it to the stack from the application which is running ZC

Understood, thanks for clarifying

nScrub for instance is doing this), but this required non-trivial changes to the application.

Thanks for the reference. Regardless of whether I end up sinking time into exploring or implementing a solution, I enjoy reading about other applications using ZC for anything other than the "capture packets" use-case, so it's appreciated either way

So at the least, my rx thread would need to also handle packets that are not intended for the application. Those would then need to be pushed into the "stack" module, to get them through to the kernel tcp/ip stack to the SSH daemon, I think. At least for the ingress delivery. I'm not sure how the return traffic would work yet but I have enough to look into it

Even it's not really a solution for my use-case, it would be neat to have a PF_RING ZC equivalent project to the libvma project that Mellanox has on top of Infiniband verbs. I know, it's far from apples to apples, but similar enough

I'm sure you're familiar with IB verbs, but if you're not familiar with libvma- in a nutshell, it's a shim (drop in replacement, really) for the standard high-level socket libc functions that's implemented as a shared library. So it can be "enabled" for any libc-based application, without any changes to the executable, via LD_PRELOAD. Lots of caveats, but it's a great approach. It doesn't help the hip rust & golang crowd though, since they don't link against libc

Anyway, thanks for the sanity check and guidance!

[cardigliano]

Yes, you should use the pf_ring "stack" module to both inject (pfring_send) traffic to the stack and read (pfring_recv) and send to the wire traffic from the stack.

[cardigliano]

And thank you for the libvma references and explanation, I know similar projects and it's really interesting.