How to set up jumbo frames support in bess?

[vsag96]

I set the mtu in bess configuration script as follows. If I increase the mtu, I get an error saying mtu between 68 and 2048 are the only supported values.

name="mlx5_nic"
nic = PMDPort(name=name,pci=pci,num_inc_q=num_queues,num_out_q=num_queues)
nic.set_port_config(name=name,mtu=2048)

However when I edit the core/snbuf_layout.h to change SNBUF_DATA 9216. Bess compiles but I get the following error when I try to start the daemon. I have 30 1G huge pages allocated and 26 of the 30 are free. ulimit -l says unlimited. What changes should I make for bess to support Jumbo frames? Thanks for your help.

Troubleshooting:
  - Check 'ulimit -l'
  - Do you have enough memory on the machine?
  - Maybe memory is too fragmented. Try rebooting.
F1108 22:40:38.774577 20079 debug.cc:405] Backtrace (recent calls first) ---
(0): /home/vthape2/bess/core/bessd(_ZN4bess10PacketPool12PostPopulateEv+0x1a5) [0x560be94e1f55]
    bess::PacketPool::PostPopulate() at /home/vthape2/bess/core/packet_pool.cc:159
         156: 
         157:   LOG(INFO) << name_ << " has been created with " << Capacity() << " packets";
         158:   if (Capacity() == 0) {
      -> 159:     LOG(FATAL) << name_ << " has no packets allocated\n"
         160:                << "Troubleshooting:\n"
         161:                << "  - Check 'ulimit -l'\n"
         162:                << "  - Do you have enough memory on the machine?\n"
(1): /home/vthape2/bess/core/bessd(_ZN4bess14DpdkPacketPoolC1Emi+0x11d) [0x560be94e232d]
    bess::DpdkPacketPool::DpdkPacketPool(unsigned long, int) at /home/vthape2/bess/core/packet_pool.cc:239
      -> 239:   PostPopulate();
(2): /home/vthape2/bess/core/bessd(_ZN4bess10PacketPool18CreateDefaultPoolsEm+0xe9) [0x560be94e2959]
    bess::PacketPool::CreateDefaultPools(unsigned long) at /home/vthape2/bess/core/packet_pool.cc:51
      -> 51:       default_pools_[sid] = new DpdkPacketPool(capacity, sid);
(3): /home/vthape2/bess/core/bessd(main+0x22e) [0x560be9400ace]
    main at /home/vthape2/bess/core/main.cc:86
      -> 86:   bess::PacketPool::CreateDefaultPools(FLAGS_buffers);
(4): /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe6) [0x7f5aafc81b96]
    __libc_start_main at ??:?
(5): /home/vthape2/bess/core/bessd(_start+0x29) [0x560be94dfff9]
    _start at ??:?

[vsag96]

Editing this comment.

The previous edit, I made some wrong remarks. The problem is rte_mempool_populate_default returns a negative value even though there is enough memory. I am not sure why this happens. I will post it here once I figure that out.

[wjuni]

I think updating buffer size from 262144 to smaller value (e.g. 65536) at core/opts.cc:127 would work.

---

For someone who wants to enable jumbo frame support with PMD port and VPort, I think the following steps would do:

1. As mentioned before, update the value of SNBUF_DATA at core/snbuf_layout.h:59 to some value multiple of 64. (e.g. 9216)
2. Update the buffer size from 262144 to smaller value power of 2 (e.g. 65536) at core/opts.cc:127
3. For VPort, add the following to core/kmod/sn_netdev.c:827.

   netdev->max_mtu=9000;
   netdev->mtu=9000;

4. For PMD port, either use configuration script nic.set_port_config(mtu=9000) or add following to core/drivers/pmd.cc:323

   rte_eth_dev_set_mtu(ret_port_id, 9000);

[vsag96]

You can also allocate more than 1024 Huge pages core/opts.cc I missed this comment for some reason. Does increasing the # huge pages hinder performance?