Request for documentation / information

[gshanemiller]

Can Amazon provide the following to aid with DPDK development,

• A NIC Datasheet. For example, Intel provides a nice data sheet with detailed descriptions of some important values. See for example pg. 494 where it describes WTHRESH, PTHRESH, and HTHRESH values
• A better description of the threshold fields here. The DPDK one liner in the code doesn't help me much. Again, as DPDK has Intel as an important contributor, a lot of the performance code in the DPDK is optimized for Intel NICs. We need to know what works for ENA NICs,
• Can you recommend a setting for tx_rs_thresh here
• The line speed for an ENA NIC seems unknowable and unsettable. Both rte_eth_link_get_nowait. and rte_eth_link_get report 0x00 which does not correspond to RTE_ETH_SPEED_NUM_ define not RTE_ETH_SPEED_NUM_NONE. After some further testing via DPDK's dpdk-testpmd task, there seems to be no way to set or get a link speed. What speed then does the ENA NIC default to?

Elsewhere, for example during the EC2 launch screen for c5n.metal instances, the web pages says c5n.metal instances are capable of 100Gbp with EFA enabled:

This instance type supports the Elastic Fabric Adapter (EFA). EFAs support OS-bypass functionality, which enables High
Performance Computing (HPC) and Machine Learning (ML) applications to communicate directly with the network interface
hardware. EFAs provide lower latency and higher throughput than traditional TCP channels.

Was that performance achieved on this driver?

[shaibran]

Let us review your request and we'll get back to you shortly

[shaibran]

Regarding documentation:

• ENA uses only the standard DPDK features. See DPDK features documentation . If the API documentation is missing, it's very unlikely that ENA will make use of it. This is DPDK API and its documentation is out of ENA scope. We suggest to request a better documentation of this API directly from the DPDK community.
• We do document all non-standard behaviors for the ENA driver and we're constantly working on the ENA DPDK README to provide best practices, troubleshooting and FAQs.
• We'll review the datasheet reference you provided and see what equivalent ENA data we can expose in order to improve app developers user experience.

Regarding threshold recommendation: We're not using any of the threshold registers in our PMD.

Regarding line speed: ENA is a virtual device, so there is no link. Thus the default link speed for the ENA is RTE_ETH_SPEED_NUM_NONE. The maximum performance is limited by the VF's capabilities and the instance type. As the driver is not aware on which instance type it is running, using appropriate limiters for the instance should be performed on the application side, as it can get access to more variety of tools to do that (e.g. it can read the instance meta data or pass the maximum BW as an argument).

Regarding performance results: You can find the sample results for the c5n.18xlarge. It was achieved by using the benchmarking tool based on the pktgen and the DTS.

[gshanemiller]

Thank you - all questions addressed.

[gshanemiller]

Reopening this item with one further request for clarification. As per

https://www.amazon.com/Data-Plane-Development-Kit-DPDK/dp/0367373955/ref=sr_1_1?keywords=data+plane+development+kit&qid=1648425589&s=books&sprefix=data+plane+devel%2Cstripbooks%2C85&sr=1-1

virtio devices (see chapter 12 pg 236) are partially or wholly interrupt driven (INTx/MSI-X).

And as per this link the number one performance constraint on the TX side is whether or not the TXQ's ring is full. As you can see code spends a lot of CPU time busy-spinning on rte_eth_tx_burst trying to flush packets out once the TXQ's ring is full.

which leads to this question:

• is the user mode virtio driver provided in this repo interrupt driven? If so how is the ENA driver using it poll driven which is what one would naively expect per DPDK doc? Not sure of details but an interrupt-rate-bouned-pathway surely has to be slower than one which is 100% poll driven. Maybe, in a cloud virtualized world, there is no other way ... any context would be appreciated.

[shaibran]

By default, the ENA PMD works fully in the in the polling mode. The cleanup is performed right after transmit (since v22.03 it's right before the transmit). Regarding As you can see code spends a lot of CPU time busy-spinning on rte_eth_tx_burst trying to flush packets out once the TXQ's ring is full --> That's because application is pushing the packets faster than the hardware can handle it

We support only Rx interrupt, however it is optional (disabled by default). This is application's power-saving feature, you can find an example for usage in the l3fwd-power application