search

ntop / PF_RING

High-speed packet processing framework
http://www.ntop.org
GNU Lesser General Public License v2.1
2.64k stars 351 forks source link

Using multiple VM instances with pfring_zc #582

Open tsivakca67 opened 4 years ago

tsivakca67 commented 4 years ago

I had trouble receiving packets on multiple instances of vm. So to create a test case, I am trying out the sample programs found in example_zc

I have spinned out 2 identical vm's in the host machine with the monitor ports being qmp1 and qmp2 ( inserted "Uio and Uio_ivshmem.ko modules)

On the host machine executing the following command ./zbalance_ipc -I eth0 -n 6 -m 0 -c 100 -Q /tmp/qmp1,/tmp/qmp2

In the guest vm's I am executing the following command ./zcount_ipc -I 0 -c 100 in one vm and ./zcount_ipc -I 1 -c 100 in another vm I am only able to receive incoming packets in one of the vm's

on another vm I am getting the pfring_zc_ipc_attach_queue error( [No buffer space available] Please check that cluster 100 is running)

I have tried mulitple instances with more vm's but find only one vm instance is working.

The following is what I have observed on the vm's. On a working vm instance, I find both /dev/uio0 and /dev/uio1 being dynamically created

whereas all other vm instances, I find only /dev/uio0 being created.

tsivakca67 commented 4 years ago

More details for the above issue. ---------------------------------------Kernel messages on a non-working instance ------- [root@guest2 ~]# dmesg [ 1378.964552] pci 0000:00:08.0: [1af4:1003] type 00 class 0x078000 [ 1378.964629] pci 0000:00:08.0: reg 0x10: [io 0x0000-0x001f] [ 1378.964665] pci 0000:00:08.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 1378.965319] pci 0000:00:08.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff] [ 1378.966049] pci 0000:00:08.0: BAR 0: assigned [io 0x1000-0x101f] [ 1378.966745] virtio-pci 0000:00:08.0: enabling device (0000 -> 0003) [ 1379.032321] virtio-pci 0000:00:08.0: virtio_pci: leaving for legacy driver [ 1379.034588] virtio-pci 0000:00:08.0: irq 39 for MSI/MSI-X [ 1379.034608] virtio-pci 0000:00:08.0: irq 40 for MSI/MSI-X [ 1379.096018] pci_bus 0000:01: Allocating resources [ 1379.109586] pci 0000:00:0a.0: [1af4:1110] type 00 class 0x050000 [ 1379.109675] pci 0000:00:0a.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 1379.109724] pci 0000:00:0a.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 1379.109798] pci 0000:00:0a.0: reg 0x18: [mem 0x00000000-0x0fffffff 64bit pref] [ 1379.110246] pci 0000:00:0a.0: BAR 2: assigned [mem 0x440000000-0x44fffffff 64bit pref] [ 1379.112677] pci 0000:00:0a.0: BAR 1: assigned [mem 0xc0001000-0xc0001fff] [ 1379.114973] pci 0000:00:0a.0: BAR 0: assigned [mem 0xc0002000-0xc00020ff] [root@guest2 ~]# ls -l /dev/uio* crw-------. 1 root root 242, 0 May 1 01:42 /dev/uio0 Note: udevd has not created /dev/uio0 here. [root@guest2 ~]# /opt/purescale/bin/zcount_ipc -i 0 -c 103 -u pfring_zc_ipc_attach_queue error [No buffer space available]

----------------------------------Working instance ---------------------------------------- [root@guest4 ~]# dmesg [ 1357.002205] pci 0000:00:0e.0: [1af4:1003] type 00 class 0x078000 [ 1357.002284] pci 0000:00:0e.0: reg 0x10: [io 0x0000-0x001f] [ 1357.002321] pci 0000:00:0e.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 1357.002970] pci 0000:00:0e.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff] [ 1357.003721] pci 0000:00:0e.0: BAR 0: assigned [io 0x1000-0x101f] [ 1357.004445] virtio-pci 0000:00:0e.0: enabling device (0000 -> 0003) [ 1357.053359] virtio-pci 0000:00:0e.0: virtio_pci: leaving for legacy driver [ 1357.086639] virtio-pci 0000:00:0e.0: irq 47 for MSI/MSI-X [ 1357.086660] virtio-pci 0000:00:0e.0: irq 48 for MSI/MSI-X [ 1357.133137] pci_bus 0000:01: Allocating resources [ 1357.253585] pci 0000:00:0f.0: [1af4:1110] type 00 class 0x050000 [ 1357.253674] pci 0000:00:0f.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 1357.253723] pci 0000:00:0f.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 1357.253798] pci 0000:00:0f.0: reg 0x18: [mem 0x00000000-0x0fffffff 64bit pref] [ 1357.254398] pci 0000:00:0f.0: BAR 2: assigned [mem 0x450000000-0x45fffffff 64bit pref] [ 1357.257616] pci 0000:00:0f.0: BAR 1: assigned [mem 0xc0001000-0xc0001fff] [ 1357.260784] pci 0000:00:0f.0: BAR 0: assigned [mem 0xc0002000-0xc00020ff] [ 1357.364506] pci_bus 0000:01: Allocating resources

[root@guest4 ~]# ls -l /dev/uio* crw-------. 1 root root 238, 0 May 1 01:40 /dev/uio0 crw-------. 1 root root 238, 1 May 1 01:42 /dev/uio1 Pls note that udevd has created 2 device nodes here

[root@guest4 bin]# ./zcount_ipc -i 1 -c 103 -u

Absolute Stats: 0 pkts (0 drops) - 0 bytes

tsivakca67 commented 4 years ago

Versions used: pfring (7.3.0) qemu (2.5.1.1)

cardigliano commented 4 years ago

Please run zbalance_ipc with -E (debug mode) and provide the output. It would be useful to check also the QEMU logs.

tsivakca67 commented 4 years ago

Hi Cardigliano, Apologize for the delay as I could not get this sent earlier to you.

The zbalance_ipc does not have -E switch so I collected other information from pfring driver and kernel messages.
We might need your help to fix this issue. Let me know if you need any other logs and the method to collect them to help you analyze the issue.

Thank you, Siva

Host: (Firing zbalance_ipc command) sh-4.2# ./zbalance_ipc -i psdeth0,psdeth1 -n 4 -c 100 -v -Q /tmp/qmp1,/tmp/qmp2 19/May/2020 05:51:39 [zbalance_ipc.c:1034] Starting balancer with 4 consumer queues.. 19/May/2020 05:51:39 [zbalance_ipc.c:1044] Run your application instances as follows: 19/May/2020 05:51:39 [zbalance_ipc.c:1049] pfcount -i zc:100@0 19/May/2020 05:51:39 [zbalance_ipc.c:1049] pfcount -i zc:100@1 19/May/2020 05:51:39 [zbalance_ipc.c:1049] pfcount -i zc:100@2 19/May/2020 05:51:39 [zbalance_ipc.c:1049] pfcount -i zc:100@3 19/May/2020 05:51:40 [zbalance_ipc.c:265] =========================

Driver (pfring.ko) messages on the host after zbalance_ipc [3186715.853293] [PF_RING][DEBUG] ring_create:4243 [pid=32333] [3186715.853307] [PF_RING][DEBUG] ring_insert:1781 ring_insert [3186715.853309] [PF_RING][DEBUG] lockless_list_add:530 BEGIN [total=0] [3186715.853310] [PF_RING][DEBUG] lockless_list_add:557 END [total=1][id=0][top_element_id=0] [3186715.853311] [PF_RING][DEBUG] lockless_list_add:561 [slot 0 is full] [3186715.853319] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-none.1 [3186715.853320] [PF_RING][DEBUG] ring_create:4318 created [3186715.853326] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=107) [3186715.853327] [PF_RING][DEBUG] ring_setsockopt:6869 --> SO_RING_BUCKET_LEN=1536 [3186715.853329] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=127) [3186715.853340] [PF_RING][DEBUG] ring_bind:5244 ring_bind() called [3186715.853341] [PF_RING][DEBUG] ring_bind:5259 searching device psdeth0 [3186715.853344] [PF_RING][DEBUG] packet_ring_bind:5199 packet_ring_bind(psdeth0, bucket_len=1536) called [3186715.853346] [PF_RING][DEBUG] ring_proc_remove:1009 Removing /proc/net/pf_ring/32333-none.1 [3186715.853348] [PF_RING][DEBUG] ring_proc_remove:1013 Removed /proc/net/pf_ring/32333-none.1 [3186715.853350] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-psdeth0.1 [3186715.853352] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=108) [3186715.853354] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 0 [3186715.853355] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 1 [3186715.853356] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 2 [3186715.853357] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 3 [3186715.853358] [PF_RING][DEBUG] ring_setsockopt:6571 [channel_id_mask=FFFFFFFFFFFFFFFF] [3186715.860346] [PF_RING][DEBUG] ring_mmap:5319 called [3186715.860349] [PF_RING][DEBUG] ring_mmap:5328 called, size: 4096 bytes [bucket_len=1536] [3186715.860351] [PF_RING][DEBUG] ring_alloc_mem:1699 ring_alloc_mem(bucket_len=1536) [3186715.862759] [PF_RING][DEBUG] ring_alloc_mem:1740 successfully allocated 6500352 bytes at 0xffffc90032f01000 [3186715.862760] [PF_RING][DEBUG] ring_alloc_mem:1760 allocated 4098 slots [slot_len=1584][tot_mem=6500352] [3186715.862762] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1584][tot_slots=4098] for ring on device psdeth0 [3186715.862764] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=4096, ptr=ffffc90032f01000 [3186715.862766] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186715.862780] [PF_RING][DEBUG] ring_mmap:5319 called [3186715.862781] [PF_RING][DEBUG] ring_mmap:5328 called, size: 6500352 bytes [bucket_len=1536] [3186715.862782] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1584][tot_slots=4098] for ring on device psdeth0 [3186715.862784] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=6500352, ptr=ffffc90032f01000 [3186715.862996] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186715.863003] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(179) [3186715.863009] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(182) [3186715.863017] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(184) [3186715.863022] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=124) [3186715.863217] [PF_RING][DEBUG] ring_release:4995 called ring_release(psdeth0) [3186715.863219] [PF_RING][DEBUG] ring_proc_remove:1009 Removing /proc/net/pf_ring/32333-psdeth0.1 [3186715.863221] [PF_RING][DEBUG] ring_proc_remove:1013 Removed /proc/net/pf_ring/32333-psdeth0.1 [3186715.863223] [PF_RING][DEBUG] ring_remove:1814 ring_remove() [3186715.863224] [PF_RING][DEBUG] ring_remove:1828 Found socket to remove [3186715.863225] [PF_RING][DEBUG] lockless_list_remove:584 BEGIN [total=1] [3186715.863227] [PF_RING][DEBUG] lockless_list_remove:603 END [total=0][top_element_id=0] [3186715.863228] [PF_RING][DEBUG] ring_remove:1847 leaving ring_remove() [3186715.964354] [PF_RING][DEBUG] ring_release:5152 ring_release: done [3186715.964383] [PF_RING][DEBUG] ring_create:4243 [pid=32333] [3186715.964387] [PF_RING][DEBUG] ring_insert:1781 ring_insert [3186715.964389] [PF_RING][DEBUG] lockless_list_add:530 BEGIN [total=0] [3186715.964391] [PF_RING][DEBUG] lockless_list_add:557 END [total=1][id=0][top_element_id=0] [3186715.964393] [PF_RING][DEBUG] lockless_list_add:561 [slot 0 is full] [3186715.964400] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-none.2 [3186715.964402] [PF_RING][DEBUG] ring_create:4318 created [3186715.964411] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=135) [3186715.964414] [PF_RING][DEBUG] create_cluster_referee:4674 attempting to create a referee for cluster 100 [3186715.964416] [PF_RING][DEBUG] create_cluster_referee:4688 new cluster referee created for cluster 100 [3186715.964418] [PF_RING][DEBUG] create_cluster_referee:4707 cluster 100 found or created [3186715.964420] [PF_RING][DEBUG] ring_setsockopt:7018 SO_CREATE_CLUSTER_REFEREE done [100] [3186715.964431] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=110) [3186716.548418] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.548424] [PF_RING][DEBUG] publish_cluster_object:4804 object 0.0 published in cluster 100 [3186716.548426] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [0.0@100] [3186716.548447] [PF_RING][DEBUG] ring_create:4243 [pid=32333] [3186716.548451] [PF_RING][DEBUG] ring_insert:1781 ring_insert [3186716.548452] [PF_RING][DEBUG] lockless_list_add:530 BEGIN [total=1] [3186716.548453] [PF_RING][DEBUG] lockless_list_add:557 END [total=2][id=1][top_element_id=1] [3186716.548454] [PF_RING][DEBUG] lockless_list_add:561 [slot 0 is full] [3186716.548455] [PF_RING][DEBUG] lockless_list_add:561 [slot 1 is full] [3186716.548462] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-none.3 [3186716.548463] [PF_RING][DEBUG] ring_create:4318 created [3186716.548467] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=107) [3186716.548468] [PF_RING][DEBUG] ring_setsockopt:6869 --> SO_RING_BUCKET_LEN=1518 [3186716.548469] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=127) [3186716.548474] [PF_RING][DEBUG] ring_bind:5244 ring_bind() called [3186716.548475] [PF_RING][DEBUG] ring_bind:5259 searching device psdeth0 [3186716.548478] [PF_RING][DEBUG] packet_ring_bind:5199 packet_ring_bind(psdeth0, bucket_len=1518) called [3186716.548479] [PF_RING][DEBUG] ring_proc_remove:1009 Removing /proc/net/pf_ring/32333-none.3 [3186716.548482] [PF_RING][DEBUG] ring_proc_remove:1013 Removed /proc/net/pf_ring/32333-none.3 [3186716.548484] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-psdeth0.3 [3186716.548486] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=108) [3186716.548487] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 0 [3186716.548488] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 1 [3186716.548489] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 2 [3186716.548490] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 3 [3186716.548491] [PF_RING][DEBUG] ring_setsockopt:6571 [channel_id_mask=FFFFFFFFFFFFFFFF] [3186716.554299] [PF_RING][DEBUG] ring_mmap:5319 called [3186716.554301] [PF_RING][DEBUG] ring_mmap:5328 called, size: 4096 bytes [bucket_len=1518] [3186716.554303] [PF_RING][DEBUG] ring_alloc_mem:1699 ring_alloc_mem(bucket_len=1518) [3186716.555623] [PF_RING][DEBUG] ring_alloc_mem:1740 successfully allocated 6434816 bytes at 0xffffc90039055000 [3186716.555625] [PF_RING][DEBUG] ring_alloc_mem:1760 allocated 4098 slots [slot_len=1568][tot_mem=6434816] [3186716.555627] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1568][tot_slots=4098] for ring on device psdeth0 [3186716.555629] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=4096, ptr=ffffc90039055000 [3186716.555631] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186716.555640] [PF_RING][DEBUG] ring_mmap:5319 called [3186716.555641] [PF_RING][DEBUG] ring_mmap:5328 called, size: 6434816 bytes [bucket_len=1518] [3186716.555642] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1568][tot_slots=4098] for ring on device psdeth0 [3186716.555643] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=6434816, ptr=ffffc90039055000 [3186716.555813] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186716.555817] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=140) [3186716.555818] [PF_RING][DEBUG] is_netdev_promisc:4930 checking promisc for psdeth0 [3186716.555819] [PF_RING][DEBUG] set_netdev_promisc:4944 setting promisc for psdeth0 [3186716.555825] device psdeth0 entered promiscuous mode [3186716.555893] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(179) [3186716.555895] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(182) [3186716.555901] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(184) [3186716.555911] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=110) [3186716.555913] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=126) [3186716.555915] [PF_RING][DEBUG] ring_setsockopt:6618 SO_SET_LINK_DIRECTION [pfr->mode=RX only][mode=RX only] [3186716.555917] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=111) [3186716.555918] [PF_RING][DEBUG] ring_setsockopt:6604 SO_SET_PACKET_DIRECTION [pfr->direction=RX only][direction=RX only] [3186716.555919] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=111) [3186716.555920] [PF_RING][DEBUG] ring_setsockopt:6604 SO_SET_PACKET_DIRECTION [pfr->direction=RX only][direction=RX only] [3186716.555922] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(184) [3186716.555923] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=106) [3186716.555924] [PF_RING][DEBUG] ring_setsockopt:6792 SO_ACTIVATE_RING [3186716.555934] [PF_RING][DEBUG] ring_create:4243 [pid=32333] [3186716.555936] [PF_RING][DEBUG] ring_insert:1781 ring_insert [3186716.555937] [PF_RING][DEBUG] lockless_list_add:530 BEGIN [total=2] [3186716.555938] [PF_RING][DEBUG] lockless_list_add:557 END [total=3][id=2][top_element_id=2] [3186716.555939] [PF_RING][DEBUG] lockless_list_add:561 [slot 0 is full] [3186716.555940] [PF_RING][DEBUG] lockless_list_add:561 [slot 1 is full] [3186716.555941] [PF_RING][DEBUG] lockless_list_add:561 [slot 2 is full] [3186716.555944] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-none.4 [3186716.555945] [PF_RING][DEBUG] ring_create:4318 created [3186716.555948] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=107) [3186716.555949] [PF_RING][DEBUG] ring_setsockopt:6869 --> SO_RING_BUCKET_LEN=1518 [3186716.555951] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=127) [3186716.555953] [PF_RING][DEBUG] ring_bind:5244 ring_bind() called [3186716.555954] [PF_RING][DEBUG] ring_bind:5259 searching device psdeth1 [3186716.555956] [PF_RING][DEBUG] packet_ring_bind:5199 packet_ring_bind(psdeth1, bucket_len=1518) called [3186716.555957] [PF_RING][DEBUG] ring_proc_remove:1009 Removing /proc/net/pf_ring/32333-none.4 [3186716.555959] [PF_RING][DEBUG] ring_proc_remove:1013 Removed /proc/net/pf_ring/32333-none.4 [3186716.555960] [PF_RING][DEBUG] ring_proc_add:990 Added /proc/net/pf_ring/32333-psdeth1.4 [3186716.555962] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=108) [3186716.555963] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 0 [3186716.555964] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 1 [3186716.555965] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 2 [3186716.555966] [PF_RING][DEBUG] ring_setsockopt:6557 Setting channel 3 [3186716.555967] [PF_RING][DEBUG] ring_setsockopt:6571 [channel_id_mask=FFFFFFFFFFFFFFFF] [3186716.564296] [PF_RING][DEBUG] ring_mmap:5319 called [3186716.564298] [PF_RING][DEBUG] ring_mmap:5328 called, size: 4096 bytes [bucket_len=1518] [3186716.564300] [PF_RING][DEBUG] ring_alloc_mem:1699 ring_alloc_mem(bucket_len=1518) [3186716.565773] [PF_RING][DEBUG] ring_alloc_mem:1740 successfully allocated 6434816 bytes at 0xffffc90039679000 [3186716.565774] [PF_RING][DEBUG] ring_alloc_mem:1760 allocated 4098 slots [slot_len=1568][tot_mem=6434816] [3186716.565776] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1568][tot_slots=4098] for ring on device psdeth1 [3186716.565777] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=4096, ptr=ffffc90039679000 [3186716.565779] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186716.565788] [PF_RING][DEBUG] ring_mmap:5319 called [3186716.565789] [PF_RING][DEBUG] ring_mmap:5328 called, size: 6434816 bytes [bucket_len=1518] [3186716.565790] [PF_RING][DEBUG] ring_mmap:5374 mmap [slot_len=1568][tot_slots=4098] for ring on device psdeth1 [3186716.565791] [PF_RING][DEBUG] do_memory_mmap:5276 mode=0, size=6434816, ptr=ffffc90039679000 [3186716.565898] [PF_RING][DEBUG] ring_mmap:5423 succeeded [3186716.565901] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=140) [3186716.565902] [PF_RING][DEBUG] is_netdev_promisc:4930 checking promisc for psdeth1 [3186716.565904] [PF_RING][DEBUG] set_netdev_promisc:4944 setting promisc for psdeth1 [3186716.565908] device psdeth1 entered promiscuous mode [3186716.565943] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(179) [3186716.565945] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(182) [3186716.565949] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(184) [3186716.565953] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=110) [3186716.565955] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=126) [3186716.565956] [PF_RING][DEBUG] ring_setsockopt:6618 SO_SET_LINK_DIRECTION [pfr->mode=RX only][mode=RX only] [3186716.565958] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=111) [3186716.565959] [PF_RING][DEBUG] ring_setsockopt:6604 SO_SET_PACKET_DIRECTION [pfr->direction=RX only][direction=RX only] [3186716.565961] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=111) [3186716.565962] [PF_RING][DEBUG] ring_setsockopt:6604 SO_SET_PACKET_DIRECTION [pfr->direction=RX only][direction=RX only] [3186716.565964] [PF_RING][DEBUG] ring_getsockopt:7206 --> getsockopt(184) [3186716.565965] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=106) [3186716.565966] [PF_RING][DEBUG] ring_setsockopt:6792 SO_ACTIVATE_RING [3186716.565998] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.565999] [PF_RING][DEBUG] publish_cluster_object:4804 object 1.0 published in cluster 100 [3186716.566000] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [1.0@100] [3186716.566026] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566027] [PF_RING][DEBUG] publish_cluster_object:4804 object 1.1 published in cluster 100 [3186716.566028] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [1.1@100] [3186716.566060] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566061] [PF_RING][DEBUG] publish_cluster_object:4804 object 1.2 published in cluster 100 [3186716.566062] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [1.2@100] [3186716.566085] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566086] [PF_RING][DEBUG] publish_cluster_object:4804 object 1.3 published in cluster 100 [3186716.566087] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [1.3@100] [3186716.566092] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566093] [PF_RING][DEBUG] publish_cluster_object:4804 object 2.0 published in cluster 100 [3186716.566094] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [2.0@100] [3186716.566096] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566097] [PF_RING][DEBUG] publish_cluster_object:4804 object 2.1 published in cluster 100 [3186716.566098] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [2.1@100] [3186716.566100] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566101] [PF_RING][DEBUG] publish_cluster_object:4804 object 2.2 published in cluster 100 [3186716.566102] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [2.2@100] [3186716.566103] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566105] [PF_RING][DEBUG] publish_cluster_object:4804 object 2.3 published in cluster 100 [3186716.566106] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [2.3@100] [3186716.566107] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=136) [3186716.566108] [PF_RING][DEBUG] publish_cluster_object:4804 object 2.4 published in cluster 100 [3186716.566109] [PF_RING][DEBUG] ring_setsockopt:7032 SO_PUBLISH_CLUSTER_OBJECT done [2.4@100] [3186717.629839] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186718.629949] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186719.630079] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186720.630205] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186721.630329] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186722.630444] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186723.630560] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186724.630684] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186725.630797] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186726.630913] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186727.631025] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186728.631138] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) [3186729.631253] [PF_RING][DEBUG] ring_setsockopt:6449 --> ring_setsockopt(optname=133) The last message keeps repeating

Both Guest 1 and Guest 2 VM's have /dev/hugepages configured and mounted modprobe uio and insmod uio_ivshmem.ko commands executed

Guest 1: (dmesg after firing zbalance_ipc command on host) [root@guest1 ~]# dmesg [ 653.196937] pci 0000:00:05.0: [1af4:1003] type 00 class 0x078000 [ 653.197073] pci 0000:00:05.0: reg 0x10: [io 0x0000-0x001f] [ 653.197132] pci 0000:00:05.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 653.198073] pci 0000:00:05.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff] [ 653.199517] pci 0000:00:05.0: BAR 0: assigned [io 0x1000-0x101f] [ 653.201232] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) [ 653.235093] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 653.235734] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 653.237396] virtio-pci 0000:00:05.0: irq 31 for MSI/MSI-X [ 653.237427] virtio-pci 0000:00:05.0: irq 32 for MSI/MSI-X [ 653.244816] pci_bus 0000:01: Allocating resources [ 653.251726] pci 0000:00:06.0: [1af4:1110] type 00 class 0x050000 [ 653.251814] pci 0000:00:06.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 653.251862] pci 0000:00:06.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 653.251936] pci 0000:00:06.0: reg 0x18: [mem 0x00000000-0x0fffffff 64bit pref] [ 653.252335] pci 0000:00:06.0: BAR 2: assigned [mem 0x840000000-0x84fffffff 64bit pref] [ 653.255002] pci 0000:00:06.0: BAR 1: assigned [mem 0xc0001000-0xc0001fff] [ 653.256450] pci 0000:00:06.0: BAR 0: assigned [mem 0xc0002000-0xc00020ff]

Guest 2: (dmesg after firing zbalance_ipc command on host) [root@guest2 ~]# dmesg [ 612.701712] pci 0000:00:05.0: [1af4:1003] type 00 class 0x078000 [ 612.701838] pci 0000:00:05.0: reg 0x10: [io 0x0000-0x001f] [ 612.701897] pci 0000:00:05.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 612.702828] pci 0000:00:05.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff] [ 612.704255] pci 0000:00:05.0: BAR 0: assigned [io 0x1000-0x101f] [ 612.705952] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) [ 612.738993] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 [ 612.739638] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver [ 612.741326] virtio-pci 0000:00:05.0: irq 31 for MSI/MSI-X [ 612.741345] virtio-pci 0000:00:05.0: irq 32 for MSI/MSI-X [ 612.748751] pci_bus 0000:01: Allocating resources [ 612.755591] pci 0000:00:06.0: [1af4:1110] type 00 class 0x050000 [ 612.755678] pci 0000:00:06.0: reg 0x10: [mem 0x00000000-0x000000ff] [ 612.755727] pci 0000:00:06.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 612.755809] pci 0000:00:06.0: reg 0x18: [mem 0x00000000-0x0fffffff 64bit pref] [ 612.756213] pci 0000:00:06.0: BAR 2: assigned [mem 0x840000000-0x84fffffff 64bit pref] [ 612.757778] pci 0000:00:06.0: BAR 1: assigned [mem 0xc0001000-0xc0001fff] [ 612.759181] pci 0000:00:06.0: BAR 0: assigned [mem 0xc0002000-0xc00020ff]

Guest 1:

running zcount_ipc program [root@guest1 bin]# ./zcount_ipc -i 0 -c 100 -u

Absolute Stats: 0 pkts (0 drops) - 0 bytes

------zcount_ipc runs fine here and if traffic is present the stat count increases ---

Guest 2: running zcount_ipc program

[root@guest2 bin]# ./zcount_ipc -i 1 -c 100 -u pfring_zc_ipc_attach_queue error [No buffer space available] Please check that cluster 100 is running Segmentation fault

cardigliano commented 4 years ago

@tsivakca67 sorry I need debug info from zbalance_ipc, it seems you are using an old version, please update.

tsivakca67 commented 4 years ago

Thank you for the information. The version I was using was 7.3.0. Let me upgrade to stable-7.6.0 from GitHub and update you on the results.

tsivakca67 commented 4 years ago

The output with -E option

./zbalance_ipc -i psdeth2,psdeth3 -c 100 -n 4 -E -Q /tmp/qmp1,/tmp/qmp2

[PF_RING-ZC][DEBUG] 98376 1518-byte (1600-byte) buffers requested [PF_RING-ZC][DEBUG] 206003200 bytes approximated memory required [PF_RING-ZC][DEBUG] 268435456 bytes rounded memory required [PF_RING-ZC][DEBUG] 20480 2097152-byte pages available at /dev/hugepages [PF_RING-ZC][DEBUG] 128 2097152-byte pages requested (cluster id = 100) [PF_RING-ZC][DEBUG] 128 2097152-byte pages allocated with va = 0x2aaaaac00000 pa = [ 525349158912, 525355450368, 525353353216, 525359644672, 525357547520, 525363838976, 525361741824, 525368033280, 525365936128, 525372227584, 525370130432, 525376421888, 525374324736, 525380616192, 525378519040, 525384810496, 525382713344, 525389004800, 525386907648, 525393199104, 525391101952, 525397393408, 525395296256, 525401587712, 525399490560, 525405782016, 525403684864, 525409976320, 525407879168, 525414170624, 525412073472, 525418364928, 525416267776, 525422559232, 525420462080, 525426753536, 525424656384, 525430947840, 525428850688, 525435142144, 525433044992, 525439336448, 525437239296, 525443530752, 525441433600, 525447725056, 525445627904, 525451919360, 525449822208, 525456113664, 525454016512, 525460307968, 525458210816, 525464502272, 525462405120, 525468696576, 525466599424, 525472890880, 525470793728, 525477085184, 525474988032, 525481279488, 525479182336, 525485473792, 525483376640, 525489668096, 525487570944, 525493862400, 525491765248, 525498056704, 525495959552, 525502251008, 525500153856, 525506445312, 525504348160, 525510639616, 525508542464, 525514833920, 525512736768, 525519028224, 525516931072, 525523222528, 525521125376, 525527416832, 525525319680, 525531611136, 525529513984, 525535805440, 525533708288, 525539999744, 525537902592, 525544194048, 525542096896, 525548388352, 525546291200, 525552582656, 525550485504, 525556776960, 525554679808, 525560971264, 525558874112, 525565165568, 525563068416, 525569359872, 525567262720, 525573554176, 525571457024, 525577748480, 525575651328, 525581942784, 525579845632, 525586137088, 525584039936, 525590331392, 525588234240, 525594525696, 525592428544, 525598720000, 525596622848, 525602914304, 525600817152, 525607108608, 525605011456, 525611302912, 525609205760, 525615497216, 525613400064, 525619691520 ] [PF_RING-ZC][DEBUG] Initialising memory segment [PF_RING-ZC][DEBUG] Memory allocation completed successfully [PF_RING-ZC][DEBUG] Listening on socket /tmp/pfring-zc-kvm-shmem-sock-100 [PF_RING-ZC][DEBUG] Connecting to socket /tmp/qmp1 [PF_RING-ZC][DEBUG] QMP Message: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 5, "major": 2}, "package": ""}, "capabilities": []}} [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "qmp_capabilities" } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "device_del", "arguments": { "id" : "shmem-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "chardev-remove", "arguments": { "id" : "shmemsock-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "device_del", "arguments": { "id" : "vport-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "chardev-remove", "arguments": { "id" : "vportsock-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments" : { "driver": "virtio-serial" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "chardev-add", "arguments" : { "id" : "vportsock-100", "backend" : { "type" : "socket", "data" : { "addr" : { "type" : "unix", "data" : { "path" : "/tmp/pfring-zc-kvm-vport-sock-100" } }, "server" : true, "wait" : false } } } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments": { "driver" : "virtserialport", "chardev" : "vportsock-100", "name" : "pfring-zc-vport-100", "id" : "vport-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "chardev-add", "arguments" : { "id" : "shmemsock-100", "backend" : { "type" : "socket", "data" : { "addr" : { "type" : "unix", "data" : { "path" : "/tmp/pfring-zc-kvm-shmem-sock-100" } }, "server" : false, "wait" : false } } } } [PF_RING-ZC][DEBUG] SHM new connection [PF_RING-ZC][DEBUG] SHM increasing vm slots [PF_RING-ZC][DEBUG] SHM Live_vms[0] [PF_RING-ZC][DEBUG] SHM trying to send fds to new connection [PF_RING-ZC][DEBUG] SHM Connected (count = 0). [PF_RING-ZC][DEBUG] SHM Live_count is 1 [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments": { "driver" : "ivshmem", "chardev" : "shmemsock-100", "size" : "256m", "ioeventfd" : "false", "id" : "shmem-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] VM initialized successfully [PF_RING-ZC][DEBUG] Connecting to socket /tmp/pfring-zc-kvm-vport-sock-100 [PF_RING-ZC][DEBUG] Connecting to socket /tmp/qmp2 [PF_RING-ZC][DEBUG] QMP Message: {"QMP": {"version": {"qemu": {"micro": 1, "minor": 5, "major": 2}, "package": ""}, "capabilities": []}} [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "qmp_capabilities" } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "device_del", "arguments": { "id" : "shmem-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "chardev-remove", "arguments": { "id" : "shmemsock-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "device_del", "arguments": { "id" : "vport-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute": "chardev-remove", "arguments": { "id" : "vportsock-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments" : { "driver": "virtio-serial" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "chardev-add", "arguments" : { "id" : "vportsock-100", "backend" : { "type" : "socket", "data" : { "addr" : { "type" : "unix", "data" : { "path" : "/tmp/pfring-zc-kvm-vport-sock-100" } }, "server" : true, "wait" : false } } } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments": { "driver" : "virtserialport", "chardev" : "vportsock-100", "name" : "pfring-zc-vport-100", "id" : "vport-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "chardev-add", "arguments" : { "id" : "shmemsock-100", "backend" : { "type" : "socket", "data" : { "addr" : { "type" : "unix", "data" : { "path" : "/tmp/pfring-zc-kvm-shmem-sock-100" } }, "server" : false, "wait" : false } } } } [PF_RING-ZC][DEBUG] SHM new connection [PF_RING-ZC][DEBUG] SHM Live_vms[1] [PF_RING-ZC][DEBUG] SHM trying to send fds to new connection [PF_RING-ZC][DEBUG] SHM Connected (count = 1). [PF_RING-ZC][DEBUG] SHM sending fd[1] to 0. MSI-efd: [PF_RING-ZC][DEBUG] SHM Live_count is 2 [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] QMP Sending Command { "execute" : "device_add" , "arguments": { "driver" : "ivshmem", "chardev" : "shmemsock-100", "size" : "256m", "ioeventfd" : "false", "id" : "shmem-100" } } [PF_RING-ZC][DEBUG] QMP Returned success [PF_RING-ZC][DEBUG] VM initialized successfully [PF_RING-ZC][DEBUG] SHM disconnection [PF_RING-ZC][DEBUG] Connecting to socket /tmp/pfring-zc-kvm-vport-sock-100 [PF_RING-ZC][DEBUG] SHM sending kill of fd[1] to 0 23/May/2020 23:06:41 [zbalance_ipc.c:1136] Starting balancer with 4 consumer queues.. 23/May/2020 23:06:41 [zbalance_ipc.c:1146] Run your application instances as follows: 23/May/2020 23:06:41 [zbalance_ipc.c:1151] pfcount -i zc:100@0 23/May/2020 23:06:41 [zbalance_ipc.c:1151] pfcount -i zc:100@1 23/May/2020 23:06:41 [zbalance_ipc.c:1151] pfcount -i zc:100@2 23/May/2020 23:06:41 [zbalance_ipc.c:1151] pfcount -i zc:100@3 23/May/2020 23:06:42 [zbalance_ipc.c:268] ========================= 23/May/2020 23:06:42 [zbalance_ipc.c:269] Absolute Stats: Recv 0 pkts (0 drops) - Forwarded 0 pkts (0 drops) 23/May/2020 23:06:42 [zbalance_ipc.c:364] =========================

23/May/2020 23:07:05 [zbalance_ipc.c:268] ========================= 23/May/2020 23:07:05 [zbalance_ipc.c:269] Absolute Stats: Recv 0 pkts (0 drops) - Forwarded 0 pkts (0 drops) 23/May/2020 23:07:05 [zbalance_ipc.c:354] Actual Stats: Recv 0.00 pps (0.00 drops) - Forwarded 0.00 pps (0.00 drops) 23/May/2020 23:07:05 [zbalanceipc.c:364] ========================= [PF_RING-ZC][DEBUG] REF VM locked object 0 type 0 mask 0 [PF_RING-ZC][DEBUG] REF VM locked object 0 type 1 mask 1 [PF_RING-ZC][DEBUG] REF VM locked object 0 type 2 mask 1_ 23/May/2020 23:07:06 [zbalance_ipc.c:268] ========================= 23/May/2020 23:07:06 [zbalance_ipc.c:269] Absolute Stats: Recv 0 pkts (0 drops) - Forwarded 0 pkts (0 drops) 23/May/2020 23:07:06 [zbalance_ipc.c:354] Actual Stats: Recv 0.00 pps (0.00 drops) - Forwarded 0.00 pps (0.00 drops) 23/May/2020 23:07:06 [zbalance_ipc.c:364] =========================

23/May/2020 23:07:31 [zbalance_ipc.c:268] ========================= 23/May/2020 23:07:31 [zbalance_ipc.c:269] Absolute Stats: Recv 0 pkts (0 drops) - Forwarded 0 pkts (0 drops) 23/May/2020 23:07:31 [zbalance_ipc.c:354] Actual Stats: Recv 0.00 pps (0.00 drops) - Forwarded 0.00 pps (0.00 drops) 23/May/2020 23:07:31 [zbalanceipc.c:364] ========================= **[PFRING-ZC][DEBUG] REF VM locked object 0 type 0 mask 0** 23/May/2020 23:07:32 [zbalance_ipc.c:268] ========================= 23/May/2020 23:07:32 [zbalance_ipc.c:269] Absolute Stats: Recv 0 pkts (0 drops) - Forwarded 0 pkts (0 drops) 23/May/2020 23:07:32 [zbalance_ipc.c:354] Actual Stats: Recv 0.00 pps (0.00 drops) - Forwarded 0.00 pps (0.00 drops) 23/May/2020 23:07:32 [zbalance_ipc.c:364] =========================

tsivakca67 commented 4 years ago

I have marked the messages in bold when zcount_ipc was executed on guest 1 and guest 2 The first bold section was when zcount_ipc was executed in guest 1, the cmd line is "./zcount_ipc -I 0 -c 100 -u" and the second one was at the guest 2 with the command "./zcount_ipc -I 1 -c 100 -u"

tsivakca67 commented 4 years ago

Hi Alfredo Cardigliano, Can I get some links to check if my environment (running vm with appropriate parameters and zbalance_ipc program running on host) is correct? That way I could cross verify and update this ticket. I am really blocked on this issue. Any information would be of great help to move forward. Thank you, Siva

cardigliano commented 4 years ago

@tsivakca67 it seems that after both VMs have been initialized (notice the "VM initialized successfully" msg), the second VM disconnects the shared memory (notice the "SHM disconnection" msg). The reason is not clear to me atm, I need to debug this (I will try to reproduce this).

tsivakca67 commented 4 years ago

Hi Alfredo, Thank you very much for your input. Would you want any other logs or information from me. Siva