Using PF_RING_ZC with PaX (grsec) patches.

[ARMATURETechnologies]

Hi,

We're trying to use ZC mode of PF_RING driver on a PaX enabled kernel (2.9.23). I had to patch the source file drivers/intel/ixgbe/ixgbe-5.0.4-zc/src/ixgbe_ptp.c and replace ACCESS_ONCE(adapter->base_incval) = incval; with ACCESS_ONCE_RW(adapter->base_incval) = incval;

It works fine using the non-ZC mode, with ixgbe driver and pf_ring module built after the previous patch. For example, with # ./userland/examples/pcount -i eth1

The network card is using the right driver. # ethtool -i eth1 driver: ixgbe version: 5.0.4 firmware-version: 0x2b2c0001, 14.5.8

However, when I try to use the zc version, I get the following error :

# ./userland/examples_zc/zcount -i zc:eth1 <Here is the licence warning. I cannot paste it here, because it is interpreted as markdown.> pfring_zc_create_cluster error [No buffer space available] Please check that pf_ring.ko is loaded and hugetlb fs is mounted

which happens when pfring_zc_create_cluster returns NULL.

I had run ./drivers/intel/ixgbe/ixgbe-5.0.4-zc/src/load_driver.sh before, and while it seems to work fine, it gave me the following warning: Warning: 512 hugepages available, 1024 requested

hugetlbfs is mounted, as i checked afterwards: # mount nodev on /dev/hugepages type hugetlbfs (rw,relatime)

What utility could I use to help me troubleshoot this issue ? Is it due to the number of hugepages available ? Any lead here would be greatly appreciated, as we really want to use PF_RING_ZC instead of just PF_RING in order to improve our capture rate.

Thanks in advance !

Here is what I got from syslog after I loaded pf_ring with enable_debug=2. There doesn't seem to be an error, or at least, none that I can see.

kernel: [24778.987495] [PF_RING][DEBUG] ring_create:4078 [pid=11546] kernel: [24778.987504] [PF_RING][DEBUG] ring_insert:1630 ring_insert kernel: [24778.987506] [PF_RING][DEBUG] lockless_list_add:464 BEGIN [total=0] kernel: [24778.987509] [PF_RING][DEBUG] lockless_list_add:491 END [total=1][id=0][top_element_id=0] kernel: [24778.987511] [PF_RING][DEBUG] lockless_list_add:495 [slot 0 is full] kernel: [24778.987525] [PF_RING][DEBUG] ring_proc_add:861 Added /proc/net/pf_ring/11546-none.11 kernel: [24778.987527] [PF_RING][DEBUG] ring_create:4142 created kernel: [24778.987539] [PF_RING][DEBUG] ring_setsockopt:6223 --> ring_setsockopt(optname=190) kernel: [24778.987542] [PF_RING][DEBUG] ring_setsockopt:6617 SO_SELECT_ZC_DEVICE eth1 kernel: [24778.987544] [PF_RING][DEBUG] pfring_select_zc_dev:5677 eth1@0 kernel: [24778.987551] [PF_RING][DEBUG] pfring_select_zc_dev:5703 found eth1 [ffff881ff8d0da80] kernel: [24778.987556] [PF_RING][DEBUG] ring_getsockopt:6940 --> getsockopt(173) kernel: [24778.987559] [PF_RING][DEBUG] pfring_get_zc_dev:5723 eth1@0 kernel: [24778.987562] [PF_RING][DEBUG] ring_proc_remove:875 Removing /proc/net/pf_ring/11546-none.11 kernel: [24778.987566] [PF_RING][DEBUG] ring_proc_remove:879 Removed /proc/net/pf_ring/11546-none.11 kernel: [24778.987569] [PF_RING][DEBUG] pfring_get_zc_dev:5748 eth1@0 [num_bound_sockets=0][ffff881ec909c000] kernel: [24778.987572] [PF_RING][DEBUG] pfring_get_zc_dev:5773 added mapping eth1@0 [num_bound_sockets=1] kernel: [24779.022655] [PF_RING][DEBUG] ring_proc_add:861 Added /proc/net/pf_ring/11546-eth1.11 kernel: [24779.022687] [PF_RING][DEBUG] ring_mmap:5127 called kernel: [24779.022689] [PF_RING][DEBUG] ring_mmap:5136 called, size: 524288 bytes [bucket_len=128] kernel: [24779.022695] [PF_RING][DEBUG] do_memory_mmap:5084 mode=1, size=524288, ptr=ffff881f72a00000 kernel: [24779.022768] [PF_RING][DEBUG] ring_mmap:5231 succeeded kernel: [24779.022780] [PF_RING][DEBUG] ring_mmap:5127 called kernel: [24779.022782] [PF_RING][DEBUG] ring_mmap:5136 called, size: 524288 bytes [bucket_len=128] kernel: [24779.022785] [PF_RING][DEBUG] do_memory_mmap:5084 mode=1, size=524288, ptr=ffff881f72b80000 kernel: [24779.022852] [PF_RING][DEBUG] ring_mmap:5231 succeeded kernel: [24779.022858] [PF_RING][DEBUG] ring_mmap:5127 called kernel: [24779.022860] [PF_RING][DEBUG] ring_mmap:5136 called, size: 524288 bytes [bucket_len=128] kernel: [24779.022862] [PF_RING][DEBUG] do_memory_mmap:5084 mode=2, size=524288, ptr=00000000dae80000 kernel: [24779.022922] [PF_RING][DEBUG] ring_mmap:5231 succeeded kernel: [24779.023230] [PF_RING][DEBUG] ring_getsockopt:6940 --> getsockopt(174) kernel: [24779.023235] [PF_RING][DEBUG] ring_getsockopt:7145 --> SO_GET_NUM_RX_CHANNELS[eth1]=1 [zc=1/rx_channels=1][ffff881ff8d0da80] kernel: [24779.029368] [PF_RING][DEBUG] ring_getsockopt:6940 --> getsockopt(182) kernel: [24779.029400] [PF_RING][DEBUG] ring_getsockopt:6940 --> getsockopt(184) kernel: [24779.029412] [PF_RING][DEBUG] ring_setsockopt:6223 --> ring_setsockopt(optname=124) kernel: [24779.029461] [PF_RING][DEBUG] ring_setsockopt:6223 --> ring_setsockopt(optname=190) kernel: [24779.029464] [PF_RING][DEBUG] ring_setsockopt:6617 SO_SELECT_ZC_DEVICE eth1 kernel: [24779.029467] [PF_RING][DEBUG] pfring_release_zc_dev:5790 releasing eth1@0 kernel: [24779.029472] [PF_RING][DEBUG] pfring_release_zc_dev:5820 eth1@0 removed mapping [num_bound_sockets=0] kernel: [24779.050333] [PF_RING][DEBUG] ring_release:4806 called ring_release(eth1) kernel: [24779.050338] [PF_RING][DEBUG] ring_proc_remove:875 Removing /proc/net/pf_ring/11546-eth1.11 kernel: [24779.050345] [PF_RING][DEBUG] ring_proc_remove:879 Removed /proc/net/pf_ring/11546-eth1.11 kernel: [24779.050348] [PF_RING][DEBUG] ring_remove:1662 ring_remove() kernel: [24779.050349] [PF_RING][DEBUG] ring_remove:1676 Found socket to remove kernel: [24779.050352] [PF_RING][DEBUG] lockless_list_remove:518 BEGIN [total=1] kernel: [24779.050354] [PF_RING][DEBUG] lockless_list_remove:537 END [total=0][top_element_id=0] kernel: [24779.050357] [PF_RING][DEBUG] ring_remove:1695 leaving ring_remove() kernel: [24779.154339] [PF_RING][DEBUG] ring_release:4963 ring_release: done kernel: [24779.154372] [PF_RING][DEBUG] ring_create:4078 [pid=11546] kernel: [24779.154379] [PF_RING][DEBUG] ring_insert:1630 ring_insert kernel: [24779.154381] [PF_RING][DEBUG] lockless_list_add:464 BEGIN [total=0] kernel: [24779.154384] [PF_RING][DEBUG] lockless_list_add:491 END [total=1][id=0][top_element_id=0] kernel: [24779.154385] [PF_RING][DEBUG] lockless_list_add:495 [slot 0 is full] kernel: [24779.154394] [PF_RING][DEBUG] ring_proc_add:861 Added /proc/net/pf_ring/11546-none.12 kernel: [24779.154396] [PF_RING][DEBUG] ring_create:4142 created kernel: [24779.154407] [PF_RING][DEBUG] ring_setsockopt:6223 --> ring_setsockopt(optname=135) kernel: [24779.154410] [PF_RING][DEBUG] create_cluster_referee:4491 attempting to create a referee for cluster 99 kernel: [24779.154413] [PF_RING][DEBUG] create_cluster_referee:4505 new cluster referee created for cluster 99 kernel: [24779.154415] [PF_RING][DEBUG] create_cluster_referee:4524 cluster 99 found or created kernel: [24779.154417] [PF_RING][DEBUG] ring_setsockopt:6756 SO_CREATE_CLUSTER_REFEREE done [99] kernel: [24779.154425] [PF_RING][DEBUG] ring_setsockopt:6223 --> ring_setsockopt(optname=110) kernel: [24779.197236] [PF_RING][DEBUG] ring_release:4806 called ring_release(none) kernel: [24779.197242] [PF_RING][DEBUG] ring_proc_remove:875 Removing /proc/net/pf_ring/11546-none.12 kernel: [24779.197249] [PF_RING][DEBUG] ring_proc_remove:879 Removed /proc/net/pf_ring/11546-none.12 kernel: [24779.197252] [PF_RING][DEBUG] ring_remove:1662 ring_remove() kernel: [24779.197254] [PF_RING][DEBUG] ring_remove:1676 Found socket to remove kernel: [24779.197256] [PF_RING][DEBUG] lockless_list_remove:518 BEGIN [total=1] kernel: [24779.197258] [PF_RING][DEBUG] lockless_list_remove:537 END [total=0][top_element_id=0] kernel: [24779.197261] [PF_RING][DEBUG] ring_remove:1695 leaving ring_remove() kernel: [24779.306346] [PF_RING][DEBUG] ring_release:4963 ring_release: done

[cardigliano]

It seems the cluster cannot find enough huge pages, please try adding more pages with:

echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

and check the availability with:

cat /sys/devices/system/node/node*/meminfo | grep Huge

More info at http://www.ntop.org/guides/pf_ring/hugepages.html

[ARMATURETechnologies]

Thanks for your quick answer ! I've tried this, but it doesn't seem to work even if the HugePages_Total is now 1024.

# cat /sys/devices/system/node/node*/meminfo | grep Huge Node 0 AnonHugePages: 10240 kB Node 0 ShmemHugePages: 0 kB Node 0 HugePages_Total: 1024 Node 0 HugePages_Free: 1024 Node 0 HugePages_Surp: 0 Node 1 AnonHugePages: 36864 kB Node 1 ShmemHugePages: 0 kB Node 1 HugePages_Total: 1024 Node 1 HugePages_Free: 1024 Node 1 HugePages_Surp: 0

I've also tried with 4096 which gave me 2048 HugePages, still not to avail.

The following values are set to your recommended values: # cat /proc/sys/kernel/shmmax 2147483648

# ulimit -l 4194304

The driver syslog output is the same. The memory is allocated and then just released.

[cardigliano]

Update: a debug session showed that this was due to the PaX kernel not exposing /proc/self/pagemap which is needed by ZC.

[ARMATURETechnologies]

Indeed ! Thanks for your help.