Open FelixMcFelix opened 5 months ago
I did some poking around in ubench to see what a reasonable upper limit in time savings is here before embarking on a redesign of PacketMeta
, based on a very rough strawman:
TL;DR: it captures a reasonable subset of a VPC-external traffic flow: we grab a lock on the port, lookup the flowkey in the btreemap, source/dest IP is rewritten, source/dest MAC is rewritten, checksums are updated and rewritten, and in the outbound case we memcpy a pre-prepared packet header stack of length ETH + v6 + UDP + Geneve + GeneveExt
.
Roughly, perf looks like (ignoring the vague complaints about regression):
#
# before
#
Benchmarking process/ULP-FastPath/wallclock/V4-Tcp-OUT-1400B: Collecting 100 samples in estimated 5.0084 s
process/ULP-FastPath/wallclock/V4-Tcp-OUT-1400B
time: [293.83 ns 295.23 ns 296.70 ns]
change: [+1.0641% +1.7977% +2.4187%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
process/ULP-FastPath/wallclock/V4-Tcp-IN-1400B
time: [216.11 ns 217.07 ns 218.10 ns]
change: [+1.1622% +1.6947% +2.2003%] (p = 0.00 < 0.05)
Performance has regressed.
#
# after
#
Benchmarking strawman/V4-Tcp-OUT-1400B(hypothetical): Collecting 100 samples in estimated 5.0071 s (2.2M it
strawman/V4-Tcp-OUT-1400B(hypothetical)
time: [73.838 ns 74.196 ns 74.560 ns]
change: [-1.2417% -0.5123% +0.2084%] (p = 0.18 > 0.05)
No change in performance detected.
strawman/V4-Tcp-IN-1400B(hypothetical)
time: [72.621 ns 73.367 ns 74.216 ns]
change: [-0.9970% +0.0679% +1.1228%] (p = 0.91 > 0.05)
No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild
So in the best case for VPC-external traffic, Port::process
could get around a ~3X speedup on inbound packets and ~4X speedup on outbound packet processing -- assuming we can build up the transforms for UFT hit packets at least.
While it's worth knowing that there's a big gap here, the effort needed to enable this (i.e., lazy reads + dirty write tracking, knowledge of structural changes) is probably pretty substantial relative to the gain. For instance, ballpark from flamegraphs indicates:
Port::process
-- ~30% (pre-#475)mac_tx
-- ~27%next_hop
-- ~21%Packet::wrap_mblk_and_parse
-- ~13%So I guess solving this will give us a ≤1.3X speedup, whereas doing some route caching can probably get us a similar speedup with less effort. I'll puzzle this out for a while.
To process a packet, we currently read each layer and then advance the
r/wptr
fields of themblk_t
containing the packet and its headers. This leaves the original headers in place. We then determine how many bytes the complete header stack would require (Option<encap{L2,L3,L4}> + Ulp {L2, L3, L4}
), before using the existing space or allocating a new mblk to be placed at the front of the chain and copying the entire (post-transform) header set back into place.However, many of our transforms when packets are
Modified
do not alter the entire header stack, or even alter the structure of ULP packet headers. While needing to add or remove the Geneve header group is universal, VPC-internal traffic would actually require no changes to the ULP, and would in effect only modify L2 addrs of transformed traffic. VPC-external packets typically only require that we change L3 src/dest addrs or L4 ports (due to NAT/SNAT) in addition to this (and, accordingly, the checksum).It would likely be more clever to:
This might in theory be extended to
Hairpin
packets under limited circumstances, but I believe that through-traffic is the most important occurrance to capture.