Running mTCP in virtualized environment

[Alex-II]

I was wondering if mTCP is able to run in a VM without (too many extra) hurdles.

I noticed issue #178 and #142 touched on this question but it's not quite clear where mTCP stands on this today.

Cheers

[ajamshed]

mTCP/DPDK can work in VMs using SR-IOV VFs. I have not created a README file on how to set up VFs. This is one sample how-to guide to set VFs: https://doc.dpdk.org/guides/nics/intel_vf.html. Let me know if you have any follow-up questions.

[vincentmli]

I am able to run the most recent mtcp in VMware ESXi VM, I think mtcp + dpdk has been improved a lot in this area, I recall with mtcp + dpdk 2.2.x, I have to do some kind of code hacks in mtcp to achieve that, but now the only change that is needed is :

diff --git a/mtcp/src/dpdk_module.c b/mtcp/src/dpdk_module.c
index 76d6da8..6dbeb4b 100644
--- a/mtcp/src/dpdk_module.c
+++ b/mtcp/src/dpdk_module.c
@@ -79,7 +79,7 @@
  * Configurable number of RX/TX ring descriptors
  */
 #define RTE_TEST_RX_DESC_DEFAULT       128
-#define RTE_TEST_TX_DESC_DEFAULT       128
+#define RTE_TEST_TX_DESC_DEFAULT       512

I am able to sort this out is after I enable the debug in dpdk, I suggest you enable the same DPDK debug and it should tell you more information.

# grep 'DEBUG=y' dpdk/x86_64-native-linuxapp-gcc/.config
CONFIG_RTE_LIBRTE_ETHDEV_DEBUG=y

then I got:

# ./apps/example/epwget 10.1.72.99 1 -f ./apps/example/epwget.conf -N 1
Configuration updated by mtcp_setconf().
Application configuration:
URL: /
# of total_flows: 1
# of cores: 1
Concurrency: 0
---------------------------------------------------------------------------------
Loading mtcp configuration from : ./apps/example/epwget.conf
Loading interface setting
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Debug dataplane logs available - lower performance
EAL: Probing VFIO support...
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 15ad:7b0 net_vmxnet3
....................CUT................
---------------------------------------------------------------------------------
Initializing port 0... rte_eth_tx_queue_setup: Invalid value for nb_tx_desc(=128), should be: <= 4096, = 512, and a product of 1
EAL: Error - exiting with code: 1
  Cause: rte_eth_tx_queue_setup:err=-22, port=0, queueid: 0

this appears to mean the nb_tx_desc min value for vmxnet3 is 512, and the code in vmxnet3 confirms that

dpdk/lib/librte_ethdev/rte_ethdev.c

        if (nb_tx_desc > dev_info.tx_desc_lim.nb_max ||
            nb_tx_desc < dev_info.tx_desc_lim.nb_min ||
            nb_tx_desc % dev_info.tx_desc_lim.nb_align != 0) {
                RTE_PMD_DEBUG_TRACE("Invalid value for nb_tx_desc(=%hu), "
                                "should be: <= %hu, = %hu, and a product of %hu\n",
                                nb_tx_desc,
                                dev_info.tx_desc_lim.nb_max,
                                dev_info.tx_desc_lim.nb_min,
                                dev_info.tx_desc_lim.nb_align);
                return -EINVAL;
        }

dpdk/drivers/net/vmxnet3/vmxnet3_ethdev.c

        dev_info->tx_desc_lim = (struct rte_eth_desc_lim) {
                .nb_max = VMXNET3_TX_RING_MAX_SIZE,
                .nb_min = VMXNET3_DEF_TX_RING_SIZE,
                .nb_align = 1,
                .nb_seg_max = VMXNET3_TX_MAX_SEG,
                .nb_mtu_seg_max = VMXNET3_MAX_TXD_PER_PKT,
        };

dpdk/drivers/net/vmxnet3/vmxnet3_ring.h

/* Default ring size */
#define VMXNET3_DEF_TX_RING_SIZE 512
#define VMXNET3_DEF_RX_RING_SIZE 128

[vincentmli]

oh, I also need to disable hardware checksum offload in my ESXi environment VM

./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum

Asim: is it possible to make nb_txd/nb_rxd configurable in mtcp/src/dpdk_module.c to accommodate the different nb_txd/nb_rxd for VM environment for different NICs?

[ajamshed]

Thanks @vincentmli!.

I can get that passed as a command-line argument in the configure script. Can you please prepare a README.vmware file that documents these steps? You can probably submit a PR on the devel branch. I will merge it thereafter.

[vincentmli]

I created PR in https://github.com/mtcp-stack/mtcp/pull/280 to make nb_txd/nb_rxd configurable in app conf file. I could also submit a README.vmware, let me know if the nb_txd/nb_rxd code change is reasonable

[ajamshed]

@vincentmli,

Apologies for the delayed response. Thanks!! I left some comments on the PR. Can you please also submit a README.vmware file? You can probably follow a rough format of how other READMEs are written. It should have enough information that a novice user can easily set up the stack if he/she follows the guideline.