is queue in NIC (physical device) itself or main memory?

[pinggit]

term "queue" is mentioned in different context.

1. sometime it is mem/buf allocated to each CPU.
2. sometime it is buffer in the NIC.

for me the main memory and NIC sounds 2 different things. kind of confusing hence needs confirmation.

I guess it's the same queue in main memory only, queue is an allocated buf in main memory, which is shared by CPU, NIC and all processes. need confirmation.

it is also possible that queue exists in both places, there is queue in NIC, and queue in main memory for vCPU (lcore).

Queue packets processing consists in :

- queues: in a network interface
- pthreads (lcore): for packet enqueue or dequeue (packet polling)
- descriptor rings (TX/RX): for packet transfer between interface queue and host memory (mbuf)
- memory spaces (mempool)

[pinggit]

answer from laurent:

---

ANSWER (LD): A queue is a concept ... that not accurately defined. In a short, this is a buffer in which incoming packets and outgoing ones are stored.

A queue has 2 directions:

• incoming
• outgoing in order to build your queue you need:
• pointers (to keep trace of level of filling)
• memory space (to store packets)

So, for DPDK and VirtIO a queue is made of:

• two rings (1 per direction)
• a shared memory space : mbuf in hugepages memory.

But sometimes, we are also speaking about a Q for each single direction .... So, Q depends of what you are telling about:

• whole Q (made up of two buffers)
• each single direction Q (a single buffer)

Threads is a different concept. The thread is the program which is processing the Q (dequeueing packets or enqueueing packets). Thread is a programming concept (parallel processing). Indeed you have a single polling thread per Q

This is why we are most often mixing this two concepts.

Keep in mind DPDK concept:

• a "network" application running in USER MODE (no access to Kernel = 1 need PMD to get packets).
• No interruption mechanism
• Packets a polled into a "infinite loop"
  • a thread is continuously polling a queue
  • a queue (bidirectional) is storing packets that still have not been processed by DPDK application

This is why we a binding a single Thread (named LCORE in DPDK ...) onto a single CPU. CPU is loaded at 100% because the (single - due to pinning and isolation rules) running on it is continuously polling the Q ... what ever a packet has been received in the Q or not (there is no interruption mechanism that could allow DPDK thread to sleep because there is no packet in the Q - this is a per design choice).

[ldurandadomia]

term "queue" is mentioned in different context.

1. sometime it is mem/buf allocated to each CPU.
2. sometime it is buffer in the NIC.

for me the main memory and NIC sounds 2 different things. kind of confusing hence needs confirmation.

I guess it's the same queue in main memory only, queue is an allocated buf in main memory, which is shared by CPU, NIC and all processes. need confirmation.

it is also possible that queue exists in both places, there is queue in NIC, and queue in main memory for vCPU (lcore).

Queue packets processing consists in :

- queues: in a network interface
- pthreads (lcore): for packet enqueue or dequeue (packet polling)
- descriptor rings (TX/RX): for packet transfer between interface queue and host memory (mbuf)
- memory spaces (mempool)

Q (ping): is a queue an structure in MEMORY, or in NIC, or in BOTH place? A (LD); We have to make distinction between several kind of queues:

• NIC Physical Q: these Q are stored inside NIC physical memory.
• DPDK Q (for physical NIC): which are Q created by DPDK application into Host memory.
• VM queues: which are Q created in host memory by QEMU virtio.

Next you have to consider the packet flow between Q (from Physical to VM).

• packet hit the physical card and is placed into the NIC physical Q.
• using RSS calculated into the NIC, packets are transfered with DMA from NIC physical Q memory to NIC DPDK queues (in host memory). This is automatically done by the NIC card. --> due PMD driver ? I do not know.
• packets are polled in each DPDK Q by the bound CPU (more exactly by the lcore thread pinned on the CPU). This thread is part of DPDK application build with DPDK lib (contrail vrouter).
• processed packet descriptors are moved into virtio ring (vnic ring). So packet is not moved, but the pointer (descriptor) is move from a ring to another ("from a Q to another").
• then the VM is processing the packet
  • either with its own DPDK application --> no need of IRQ. DPDK app on VM is continuously polling RX virtio ring.
  • either in Kernel mode (regular IP stack) --> IRQ emulation required ==> In both cases, VM and vrouter (host DPDK application) must share a common memory area in order to share rings (virtio) and packets

[pinggit]

nice! so let me see if I caught it exactly:

• packet hit NIC, NIC place them in NIC internal queues when packet are accepted by NIC (those dropped won't be placed for example) --> NIC internal queues is the set of black squares at the bottom of the diagram.

• after RSS calculation using DMA transfer, packet are moved from "NIC internal queues" to pNIC DPDK queues. RSS is managing M (internal Q) to N (DPDK queues) packets spread. DMA is process is not consuming any CPU resources. This transfer it almost transparent for the host node. This is REAL packet copy (deep copy). This is step 1 in the diagram. --> pNIC (DPDK) queues are the "pink rings" at the bottom of the diagram.

FROM NOW DPDK APPLICATION CODE IS INVOLVED _ HERE THE DPDK VROUTER IS ENTERING IN ACTION

• pNIC (DPDK) queue --> dpdk polling thread is picking the packet into the queue (in host memory/huge page). This is step 2 in this diagram. Two scenarios (depending on packet type):

  • polling thread is also processing the packet (processing thread = polling thread).
  • polling thread is calculating a new hash. Then the polling thread place the packet into a "forwarding thread" vrouter internal Q (rings not drawn in this diagram - Contrail specific - but arrows 3 are showing this balancing). -> this is descriptor/ring/pointer copy only (shallow copy)

• fwding thread -> virtio/vNIC ring, again shollow copy, per diagram below, there is a "mapping" between them. (Virtio/vNIC rings are the white rings at the top of the diagram). This is step 4 in this diagram.

• VM process it (no step shown in this diagram)

  • VM is dpdk, then repeat above (but with is on internal rings)
  • VM is kernel, interupt based communication.

[ldurandadomia]

Yes correct. Small remark. If VM is DPDK, "VM is dpdk, then repeat above (but with is on internal rings)"

I do not think so. I guess, rings are the same :-) (virtio rings = DPDK rings) VM DPDK application is polling directly virtio rings.

[pinggit]

to close this issue:

so I'd like to carefully use these terms:

• hardware queues
• physical queues
• DPDK queues
• vNIC queues

let's talk about from packet path perpsective:

• packet hit NIC
• packet stored in NIC "hardware queue"
• packet dispathed via DMA+RSS to "dpdk queue" on the pNIC side
• packet forwarded to dpdk queue on vNIC side
• packet enters vNIC

we can say: we are not quite sure how the "hardware queue" (inside NIC) is organized, and we don't care, but we know how the DPDK queue organized. and whenever you talk about DMA and RSS, it is between pNIC "hardware queue" in NIC, and "dpdk queue" in memory

"but NIC and DPDK queues are quite the same. Because DPDK and PMD take care what you have on NIC memory is replicated in host memory in real DPDK rings."