DpdkDevice::openMultiQueues() is not NUMA aware

[SesomB]

The DpdkDevice::openMultiQueues method in the PcapPlusPlus DPDK wrapper is currently not NUMA aware. It allocates memory using hard coded socket id, which is set to 0. This behavior can lead to suboptimal performance on NUMA systems, where memory access times can vary significantly depending on the proximity of the memory to the CPU cores.

Possible Solution To address this issue, the following changes can be made:

1. Add a Private Member: Introduce a private member m_DeviceSocketId in the DpdkDevice class, which will be set in the constructor.

   int16_t m_DeviceSocketId;

2. Set m_DeviceSocketId in the Constructor: Initialize m_DeviceSocketId in the DpdkDevice constructor using rte_eth_dev_socket_id.

   m_DeviceSocketId = rte_eth_dev_socket_id(m_Id);

3. Modify DpdkDevice::initMemPool to Use m_DeviceSocketId:

   // - 
   memPool = rte_pktmbuf_pool_create(mempoolName, mBufPoolSize, MEMPOOL_CACHE_SIZE, 0, m_MBufDataSize, 0);
   // +
   memPool = rte_pktmbuf_pool_create(mempoolName, mBufPoolSize, MEMPOOL_CACHE_SIZE, 0, m_MBufDataSize, m_DeviceSocketId);

4. Modify DpdkDevice::initQueues to Use m_DeviceSocketId:

   • RX queues setup:

     // -
     int ret = rte_eth_rx_queue_setup((uint8_t) m_Id, i,m_Config.receiveDescriptorsNumber, 0, NULL, m_MBufMempool);
     // +
     int ret = rte_eth_rx_queue_setup((uint8_t) m_Id, i,m_Config.receiveDescriptorsNumber, m_DeviceSocketId, NULL, m_MBufMempool);

• TX queues setup:

  // -
  int ret = rte_eth_tx_queue_setup((uint8_t) m_Id, i,m_Config.transmitDescriptorsNumber, 0, NULL);
  //+
  int ret = rte_eth_tx_queue_setup((uint8_t) m_Id, i,m_Config.transmitDescriptorsNumber,m_DeviceSocketId, NULL);

• TX Buffers allocation - Clarity changes:

  // -
  m_TxBuffers[i] = (rte_eth_dev_tx_buffer*)rte_zmalloc_socket("tx_buffer", RTE_ETH_TX_BUFFER_SIZE(MAX_BURST_SIZE), 0, rte_eth_dev_socket_id(m_Id));
  // +
  m_TxBuffers[i] = (rte_eth_dev_tx_buffer*)rte_zmalloc_socket("tx_buffer", RTE_ETH_TX_BUFFER_SIZE(MAX_BURST_SIZE), 0, m_DeviceSocketId);

By implementing these changes, the DpdkDevice::openMultiQueues method will be NUMA aware, potentially improving performance on NUMA systems by ensuring that memory is allocated on the appropriate NUMA node.

[seladb]

@SesomB I think that's a great idea! And you already laid out the entire solution 🙂 Would you consider opening a PR with this implementation?

[SesomB]

The PR addressing this issue has been successfully merged, Closing the issue now.