Packet buffer cleared to zero.

[jcmathews]

Usecase: A forwarder application running in a VM, which receives packet from a s/w queue from the host machine and forward traffic to a vNIC, then to tap and bridge.

Topology X540-AT2--->VM(QMP)--->vNIC(virtio)---->TAP-OVS

Expected output: Traffic to be sent out of vNIC to tap interface and the bridge

Current output: Traffic is received properly, sent out via vNIC and seen on tap interface, but not on OVS brigde, when checked packet buffer, it has been cleared to zero's when sending out from vNIC.

A capture from tap interface. 23:00:28.323859 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet) Null Information, send seq 0, rcv seq 0, Flags [Command], length 242 0x0000: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................

App log [PF_RING-ZC][DEBUG] 268435456 bytes requested with id = 119 [PF_RING-ZC][DEBUG] 2048 2097152-byte pages available at /dev/hugepages [PF_RING-ZC][DEBUG] 128 2097152-byte pages allocated with va = 0x2aaaaac00000 pa = [ 14451474432, 14449377280, 14455668736, 14453571584, 2231369728, 2237661184, 14459863040, 14457765888, 14464057344, 14461960192, 14468251648, 14466154496, 14472445952, 2235564032, 2241855488, 14470348800, 14476640256, 14474543104, 14480834560, 14478737408, 14485028864, 14482931712, 2239758336, 2246049792, 14489223168, 14487126016, 14493417472, 14491320320, 14497611776, 14495514624, 14501806080, 2243952640, 2250244096, 14499708928, 14506000384, 14503903232, 14510194688, 14508097536, 14514388992, 14512291840, 2248146944, 2254438400, 14518583296, 14516486144, 14522777600, 14520680448, 14526971904, 14524874752, 14531166208, 2252341248, 2258632704, 14529069056, 14535360512, 14533263360, 14539554816, 14537457664, 14543749120, 14541651968, 2256535552, 2262827008, 14547943424, 14545846272, 14552137728, 14550040576, 14350811136, 14348713984, 14355005440, 2197815296, 2204106752, 14352908288, 14359199744, 14357102592, 14363394048, 14361296896, 14367588352, 14365491200, 2202009600, 2208301056, 14371782656, 14369685504, 14375976960, 14373879808, 14380171264, 14378074112, 14384365568, 2206203904, 2212495360, 14382268416, 14388559872, 14386462720, 14392754176, 14390657024, 14396948480, 14394851328, 2210398208, 2216689664, 14401142784, 14399045632, 14405337088, 14403239936, 14409531392, 14407434240, 14413725696, 2214592512, 2220883968, 14411628544, 14417920000, 14415822848, 14422114304, 14420017152, 14426308608, 14424211456, 2218786816, 2225078272, 14430502912, 14428405760, 14434697216, 14432600064, 14438891520, 14436794368, 14443085824, 2222981120, 2229272576, 2227175424, 2233466880, 14440988672, 14447280128, 14445182976 ] [PF_RING-ZC][DEBUG] Initialising memory segment [PF_RING-ZC][DEBUG] Memory allocation completed successfully [root@localhost ~]# cat /proc/net/pf_ring/ 4091-eth1.36 4091-none.35 dev/ info stats/ [root@localhost ~]# cat /proc/net/pf_ring/4091-eth1.36 Bound Device(s) : eth1 Active : 1 Breed : Standard Appl. Name : pfring-zc-119-eth1 Socket Mode : TX only Channel Id Mask : 0xFFFFFFFFFFFFFFFF VLAN Id : 65535 Slot Version : 17 [7.3.0] Min Num Slots : 409602 Bucket Len : 1518 Slot Len : 1568 [bucket+header] Tot Memory : 642265088 TX: Send Ok : 21708118 TX: Send Errors : 1071 [root@localhost ~]# cat /proc/net/pf_ring/4091-none.35 Bound Device(s) : Active : 0 Breed : Standard Appl. Name : pfring-zc-cluster-119 Socket Mode : RX+TX Capture Direction : RX+TX Sampling Rate : 1 Filtering Sampling Rate: 0 IP Defragment : No BPF Filtering : Disabled Sw Filt Hash Rules : 0 Sw Filt WC Rules : 0 Sw Filt Hash Match : 0 Sw Filt Hash Miss : 0 Sw Filt Hash Filtered : 0 Hw Filt Rules : 0 Poll Pkt Watermark : 128 Num Poll Calls : 0 Poll Watermark Timeout : 0 [root@localhost ~]#

[jcmathews]

Application Details We are testing a modified version of zbounce inside KVM guest.

The idea is to receive packets in the VM that arrive on the in-queue (established with monitor socket in the host: Q 0) and bounce the same on another device (vNIC) in the VM.

To achieve this the zbounce code was modified to consider the queue as the input device.

There are in effect 2 clusters here. One created by the host application(C1) and the other created by the zbounce application (C2). We are trying to move a packet received from the queue of one cluster (host:C1) to the queue of the other ( cluster(created inside VM : C2).

The packet handle that we used in packet_consumer_thread() remained as in the original code, i.e, it was associated with the VM created cluster.

i->tmpbuff = pfring_zc_get_packet_handle(zc);

We used the same packet handle to receive packet from queue belonging to C1 (we were not sure if this would work) and send/bounce it using the same handle to queue belonging to C2.

Observation: recv_pkt succeeded and we are able to see received packet data in verbose mode. send_pkt also succeeded and packets are seen egressing through the output device. However, the packet header and data is zero'd out

[cardigliano]

@jcmathews please do not mix packet handles between different clusters, as they are from different "address spaces". A packet copy is required in this case.

[sajishv]

@cardigliano ,
Is there anyway to avoid this packet copy? (Requirement is to send the packet out a virtual interface in the VM and not to another qmp socket based queue.)

Inside a VM is it possible to open a new device (for TX) and associate it with the same cluster as that was created in the HOST?

[cardigliano]

@sajishv a packet copy is mandatory when forwarding to an interface which is not a ZC queue/interface, however you should not worry about the performance, as a packet copy has less overhead than sending to a virtual interface

[sajishv]

@cardigliano , I noticed that copying is not causing as much delay but sending the copied packets to a virtual interface causes lots of drops (both forward drops in HOST zbalance application and in-queue drops inside the VM application). Do we have any benchmark figures for the performance with load balancing packets across vnics inside VM? Is there any guideline to improve the performance for such cases?

[cardigliano]

@sajishv please note that if packet transmission on the virtual interface takes time, the application does not dequeue packets from the ZC queue fast enough. In essence I believe the bottleneck is on packet transmission, not on the zbalance/queue side which are just discarding packets because the queue is full.

[jcmathews]

@jcmathews please do not mix packet handles between different clusters, as they are from different "address spaces". A packet copy is required in this case.

@cardigliano, A packet copy would ideally possess some overhead and also i have observed that with C2(cluster on VM), C2 packet handle is able to receive packets but not able to send.

Do you have any PF_RING bench-marking with vNIC using virtio driver? Can PFRING accelerate a vNIC using virtio driver ?

[cardigliano]

@jcmathews, a packet copy has some overhead, however in this case it is negligible compared to the overhead for transmitting packets to a vNIC interface as kernel and hypervisor are involved (even removing the packet copy the resulting performance would not change). In order to check the vNIC performance, you can try using our pfsend tool to transmit packets on that interface and see what is the maximum rate it is able to handle on that specific hw/vm configuration.