Trex crashed on 12 interface

[jpatidar01]

Trex v3.02

On running the trex with 12 interface and 4 cores per dual interface (i.e. c=4) . It crashed on starting the traffic.

1. I tried to many combination of memory mbuf_XX : in trex cfg file but didn't works.
2. Also tried option --mbuf-factor 0.2 , 0.8 0.5 .......

Above two fix works for some packet size but not for all ( 64, 128, 256 ....)

Is their any solution that work of all pkt sizes on c>=4 case ??

log : Error: Failed allocating mbuf for holding 256 bytes from socket 1 Try to enlarge the amount of mbufs in the configuration file '/etc/trex_cfg.yaml' mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
Error: signal 11:

traceback follows

1 0x55d7da3b6873 ./_t-rex-64(+0x2f8873) [0x55d7da3b6873] 2 0x7fd1eeff0520 /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd1eeff0520] 3 0x55d7da730894 rte_mempool_ops_get_count + 4 4 0x55d7da72fd5d rte_mempool_avail_count + 13 5 0x55d7da385efe CRteMemPool::dump(_IO_FILE) + 1246 6 0x55d7da3451ac CGenNodeStateless::alloc_node_with_vm() + 4060 7 0x55d7da33f2aa CCoreEthIFStateless::send_node(CGenNode) + 442 8 0x55d7da3ad1f2 CNodeGenerator::handle_stl_node(CGenNode, CFlowGenListPerThread) + 978 9 0x55d7da3a0dd2 CNodeGenerator::flush_file(double, double, bool, CFlowGenListPerThread*, double&) + 706 10 0x55d7da58f414 TrexStatelessDpCore::start_scheduler() + 244 11 0x55d7da504ff9 TrexDpCore::start() + 57 12 0x55d7da21bdb0 CGlobalTRex::run_in_core(unsigned char) + 452 13 0x55d7da34076b ./_t-rex-64(+0x28276b) [0x55d7da34076b] 14 0x55d7da2b83a1 ./_t-rex-64(+0x1fa3a1) [0x55d7da2b83a1] 15 0x7fd1ef042b43 /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd1ef042b43] 16 0x7fd1ef0d4a00 /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd1ef0d4a00]

[hhaim]

@jpatidar01 it seems you are doing a 64bytes test and mbuf_64 is missing, The option --mbuf-factor will enlarge all the pools in the same rate and you should have a factor more than 1 (try 3) The file option is better as you can tune the number per pool

[jpatidar01]

i tried the --mbuf-factor=2 and getting below error

number of ports : 12 max cores for 2 ports : 4 tx queues per port : 6

RX core uses TX queue number 65535 on all ports core, c-port, c-queue, s-port, s-queue, lat-queue

1 0 0 1 0 0
2 2 0 3 0 0
3 4 0 5 0 0
4 6 0 7 0 0
5 8 0 9 0 0
6 10 0 11 0 0
7 0 1 1 1 255
8 2 1 3 1 255
9 4 1 5 1 255
10 6 1 7 1 255
11 8 1 9 1 255
12 10 1 11 1 255
13 0 2 1 2 255
14 2 2 3 2 255
15 4 2 5 2 255
16 6 2 7 2 255
17 8 2 9 2 255
18 10 2 11 2 255
19 0 3 1 3 255
20 2 3 3 3 255
21 4 3 5 3 255
22 6 3 7 3 255
23 8 3 9 3 255
24 10 3 11 3 255

EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested memzone segments exceeds RTE_MAX_MEMZONE ERROR there is not enough huge-pages memory in your system EAL: Error - exiting with code: 1 Cause: Cannot init nodes mbuf pool nodes-7 Killing Scapy server... Scapy server is killed

[hhaim]

@jpatidar01 try to setup huge pages using GRUB

https://doc.dpdk.org/dts/gsg/usr_guide/setup_dut.html

see here.

By default trex script allocate 4GB per NUMA. try to allocate more @ startup and then run the script

[jpatidar01]

Yes i'm setting huge pages using GRUB only

here: GRUB_CMDLINE_LINUX="hugepagesz=1G hugepages=128 default_hugepagesz=1G "

i'm using 12 interfaces and 4 core per dual interface

All the time i'm getting this error

ERROR there is not enough huge-pages memory in your system EAL: Error - exiting with code: 1 Cause: Cannot init nodes mbuf pool nodes-19

How much huge pages need for this configuration ?