AltraMayor
commented
1 month ago The flag RTE_ETH_RSS_LEVEL_INNERMOST would be an alternative solution for this problem if NICs supported it. However, this flag is virtually not supported in DPDK 23.11:
$ grep -r RTE_ETH_RSS_LEVEL_INNERMOST dependencies/dpdk/drivers/net/
dependencies/dpdk/drivers/net/bnxt/bnxt_vnic.c: rss_level |= RTE_ETH_RSS_LEVEL_INNERMOST;
After Gatekeeper processing the incoming packets, both request packets and granted packets will be sent to Grantors using IP-in-IP encapsulation. In the extreme case, all the packets from a Gatekeeper are sent to the same Grantor sever using the same IP address pair (Gatekeeper IP, Grantor IP). However, all the packets from the same Gatekeeper server will be directed to the same lcore in Grantor server by using the default RSS over the IP addresses pair.
Even though the processing logic in Grantor is simpler than Gatekeeper, this will become an issue when the Gatekeeper servers have fast NICs (e.g., 100+ Gbps), since a single lcore in Grantor cannot process all the traffic. The closed patch #316 allows Gatekeeper to forward granted packets directly to their destinations, which can reduce the load on Grantor servers significantly. However, considering the number of Gatekeeper servers sending traffic to a Grantor is much smaller than the whole IP address space, there is still a high possibility that traffic from more than one Gatekeeper server will be processed by a single core in Grantor.
One feasible solution is to explore inner RSS in Grantor, i.e., Grantor applies RSS to the inner most IP header in the IP-in-IP encapsulated packets, thus the IP pair (client IP, real server IP) is good enough to load balance the traffic among all the GT blocks.