search

p4lang / pna

Portable NIC Architecture
Apache License 2.0
54 stars 21 forks source link

Add rationale appendix section why packets can't switch from FROM_NET to TO_NET inside main #31

Open jafingerhut opened 3 years ago

jafingerhut commented 3 years ago

and similarly not from FROM_HOST to TO_HOST.

Hardware reasons involving flow control / credits/etc. are the primary reason.

jafingerhut commented 2 years ago

AI Andy: Compare two PNA-like architectures:

(a) one like the currently defined one, in which FROM_NET packets must leave as TO_HOST packets, and in which FROM_HOST packets must leave as TO_NET packets. In such an architecture, a FROM_NET packet can leave as TO_HOST with the output port set as an Ethernet port, and then it will loop back on the host side, and come back as a FROM_HOST packet and be processed again, and unless the output port is changed to a host-side vport, it will leave as a TO_NET packet and go out an Ethernet port. The second pass through the parser, main control, and main deparser is NOT optional for such packets in this architecture.

(b) another one that is the same, except FROM_NET packets may leave as either TO_HOST or TO_NET packets, and FROM_HOST packets may leave as either TO_NET or TO_HOST packets. In such an architecture, a FROM_NET packet can lavea as TO_NET with the output port set as an Ethernet port, and it will NOT loop back on the host side, but go straight to an Ethernet port.

If a program is written by a developer assuming architecture (a), is it straightforward for a compiler to take that source code and compile it to a hardware target that works like (b)? If so, how complex is the compiler's job, and does it introduce any extra processing passes in the target device?

Similar questions for a program written for architecture (b), but compiled to a hardware target that works like (a).

Note: Another issue with a hardware target that works like (b) is that there can be long bursts of time when packets are received "at line rate" from both network ports and the host, and they are all destined for the host. Or they are all destined for network ports. Thus the buffering/queueing parts of the NIC must be designed for this somehow.

Technically the same traffic pattern could occur for a long time for a hardware target that works like (a) -- similar issues can arise with buffering/queueing parts of the chip that could cause, e.g. if all traffic from both host and network ports were destined for the host, eventually some traffic either must be lost, or the NIC->host direction must somehow be designed with enough bandwidth to accept that total rate of traffic. That last part seems unlikely to me, since most likely the host->NIC and NIC->host data paths would be equal or near equal bandwidths, just as the Ethernet port->NIC and NIC->Ethernet port would be equal or near equal bandwidths.

thomascalvert-xlnx commented 2 years ago

If a program is written by a developer assuming architecture (a), is it straightforward for a compiler to take that source code and compile it to a hardware target that works like (b)?

The compiler would have to make H2H and N2N packets recirculate through the (b) pipeline to achieve the (a) semantics. This would require some hardware support - not much if recirculation is already supported, but quite a bit if it isn't. Depending on the implementation, this may also halve the achievable packet rate/throughput since each packet needs to be processed twice.

Similar questions for a program written for architecture (b), but compiled to a hardware target that works like (a).

This would require inhibiting any side-effects of the second pass through the pipeline (making it into a no-op). I believe this can be achieved by making any logic in there conditional (i.e. wrap the entire P4 code in a big if statement).

Note: Another issue with a hardware target that works like (b) is that there can be long bursts of time when packets are received "at line rate" from both network ports and the host, and they are all destined for the host. Or they are all destined for network ports. Thus the buffering/queueing parts of the NIC must be designed for this somehow.

I don't see a fundamental difference between both architectures here. If the host path is oversubscribed, packets will need to be dropped, or the sources backpressured. If anything, architecture (a) will have more trouble coping since the N2H pipeline will need to process 2x the traffic.

In terms of hardware implementation, recirculation and the associated possibility of deadlock need to be carefully considered. It must not be possible to recirculate a packet infinitely through the datapath. One option is to not support recirculation. Another possibility is to include a "passes" counter which is increased at each pass and causes the packet to be dropped if it reaches a defined maximum.

In terms of software programmability, if the PNA is aiming to be a target for SmartNICs which offload vSwitch semantics, option (b) seems like a more natural fit: the switch abstraction processes packets from any source port and egresses them to any output port(s). The concept of RX/TX direction is more applicable to previous-generation NICs which did not expect to process significant amounts of traffic along the H2H and N2N paths.