dpdk vrouter threads numbering

[pinggit]

in here we say:

lcore 0: vhost0
lcore 1: timers
lcore 2: Interrupts
lcore 8: pkt0 (can be busy)
lcore 9: Netlink (can be busy)

but in lab capture I don't see lcore0 (vhost).

# ps -Tp `pidof contrail-vrouter-dpdk`
PID SPID TTY TIME CMD
*54490 54490 ? 02:46:12 contrail-vroute*
*54490 54611 ? 00:02:33 eal-intr-thread*
*54490 54612 ? 01:35:26 lcore-slave-1*
*54490 54613 ? 00:00:00 lcore-slave-2*
*54490 54614 ? 00:00:17 lcore-slave-8*
*54490 54615 ? 00:02:14 lcore-slave-9*
*54490 54616 ? 2-21:44:06 lcore-slave-10*
*54490 54617 ? 2-21:44:06 lcore-slave-11*
*54490 54618 ? 2-21:44:06 lcore-slave-12*
*54490 54619 ? 2-21:44:06 lcore-slave-13*
*54490 54620 ? 2-21:44:06 lcore-slave-14*
*54490 54621 ? 2-21:44:06 lcore-slave-15*
*54490 54622 ? 2-21:44:06 lcore-slave-16*
*54490 54623 ? 2-21:44:06 lcore-slave-17*
*54490 54990 ? 00:00:00 lcore-slave-9*

also the source code is not very clear

• this link does not work anymore https://github.com/Juniper/contrail-vrouter/blob/master/include/vr_dpdk.h
• the content is not clear to me:

enum {
    VR_DPDK_KNITAP_LCORE_ID = 0,
    VR_DPDK_TIMER_LCORE_ID,
    VR_DPDK_UVHOST_LCORE_ID,
    /*
     * The actual number of IO lcores depends on the number of
     * forwarding lcores.
     */
    VR_DPDK_IO_LCORE_ID,
    VR_DPDK_IO_LCORE_ID2,
    VR_DPDK_IO_LCORE_ID3,
    VR_DPDK_IO_LCORE_ID4,
    VR_DPDK_LAST_IO_LCORE_ID,
    /* [PACKET_ID..FWD_ID) lcores have TX queues, but no RX queues */
    VR_DPDK_PACKET_LCORE_ID,
    VR_DPDK_NETLINK_LCORE_ID,
    /* The actual number of forwarding lcores depends on affinity mask. */
    VR_DPDK_FWD_LCORE_ID,
};

it does not tell, for example, what is LCORE_ID 0 , 2 , etc.

anyone can clarify?

[ldurandadomia]

Hi Ping,

This is a C enumeration, starting with ID = 0. So, it gives: VR_DPDK_KNITAP_LCORE_ID ==> 0 VR_DPDK_TIMER_LCORE_ID ==> 1 VR_DPDK_UVHOST_LCORE_ID ==> 2 VR_DPDK_IO_LCORE_ID ==> 3 VR_DPDK_IO_LCORE_ID2 VR_DPDK_IO_LCORE_ID3 VR_DPDK_IO_LCORE_ID4 ==> 6 VR_DPDK_LAST_IO_LCORE_ID ==> 7 VR_DPDK_PACKET_LCORE_ID ==> 8 VR_DPDK_NETLINK_LCORE_ID ==> 9 VR_DPDK_FWD_LCORE_ID ==> 10 = first ID used for polling and forwarding cores.

VR_DPDK_KNITAP_LCORE_ID ==> 0 ==> KNI/TAP ==> Link with Linux Kernel where the vhost0 interface is running vhost0 is a tap interface between host-os and vrouter. vhost0 is the layer-3 interface used by host operating system. As part of setup, Contrail software migrates the IP configuration from ethernet port to vhost0. https://sureshkvl.gitbooks.io/opencontrail-beginners-tutorial/introspect/contrail-vrouter-agent.html KNI: Kernel Network Interface. https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html KNI is optional. By default a TAP interface is used.

VR_DPDK_TIMER_LCORE_ID ==> 1 ==> TIMER task

VR_DPDK_UVHOST_LCORE_ID ==> 2 ==> UVHOST = VirtIO backend (virtual instance connectivity interrupts handling)

By default, IO cores are not in used (disabled in C code file).

VR_DPDK_PACKET_LCORE_ID ==> 8 ==> Pkt0 interface (receive packets sent by Dataplane = DPDK vrouter to the control plane = vrouter agent -> to get a central processing, like BFD packets, or to query new flow creation)

VR_DPDK_NETLINK_LCORE_ID ==> 9 ==> Netlink interface (send packets from Control plane = vrouter agent to Dataplane = DPDK vrouter ; exemple flow table, FIB table)

Laurent

[ldurandadomia]

Sources have been moved to Tungsten repo: https://github.com/tungstenfabric/tf-vrouter/blob/master/include/vr_dpdk.h

[pinggit]

very nice explanation. thanks!

about vhost0:

1. kiran has a few words about it in CEM-9701

   
   vhost0 is a linux net_device interface which has hooks to vrouter.

The tx of vhost0 has a hook to vrouter. That is, if any packets are sent from the linux host on vhost0, it lands up in vrouter kernel module. The name of the hook is vhost_dev_xmit(). The vrouter kernel module then decides what it has to do with the packet. It can choose to send that packet on the NIC if needed.

So there is no direct connection between the NIC driver and vhost0.

Similarly, on the rx side, vrouter registers a handler for all NIC interfaces (eg: eth1). So linux delivers all packets which the NIC receives to vrouter instead of the linux networking stack. Once vrouter receives the packet from the NIC, it can choose to send that to vhost0 by setting the skb->dev to vhost0 and then doing a netif_receive_skb() call.

2.  I'm not sure if above details apply to both kernel and dpdk vrouter, or put another way, if with dpdk-vrouter one process is bound to it (processing vhost0 packet), what was bound to it from kernel vrouter? a special "handler"?

[pinggit]

back to my original question: where is lcore 0 ? I don't see it in any captures. shouldn't it be something like "lcore-slave-0" ?

I see 2 lcore9, but no lcore0, is it a bug (typo)? @kirankn80 maybe you can confirm.

$ ps -Tp pidof contrail-vrouter-dpdk PID SPID TTY TIME CMD 3638 3638 ? 00:10:07 contrail-vroute #<---main process 3638 4026 ? 00:00:06 eal-intr-thread #<---control thread 3638 4027 ? 00:13:09 lcore-slave-1 #<---service thread 3638 4028 ? 00:00:00 lcore-slave-2 #<---service thread 3638 4029 ? 00:00:11 lcore-slave-8 #<---service thread 3638 4030 ? 00:01:01 lcore-slave-9 #<---service thread 3638 5941 ? 00:00:01 lcore-slave-9 #<---service thread 3638 4031 ? 07:37:41 lcore-slave-10 #<---forwarding thread 3638 4032 ? 07:37:41 lcore-slave-11 #<---forwarding thread 3638 4033 ? 07:37:41 lcore-slave-12 #<---forwarding thread 3638 4034 ? 07:37:38 lcore-slave-13 #<---forwarding thread 3638 4035 ? 07:37:41 lcore-slave-14 #<---forwarding thread 3638 4036 ? 07:37:41 lcore-slave-15 #<---forwarding thread 3638 4037 ? 07:37:41 lcore-slave-16 #<---forwarding thread 3638 4038 ? 07:37:41 lcore-slave-17 #<---forwarding thread

[pinggit]

this is from kiran:

I kind of remember this now. And this is a good question. The KNI core is the same as master core.

“3638 3638 ? 00:10:07 contrail-vroute #<---main process”

Not sure why we overloaded master core with KNI core. Maybe because master core does not do much. We can separate both of them and run perf to check what is the difference.

[pinggit]

@kirankn80 @ldurandadomia

1. regarding this 3638 3638 ? 00:10:07 contrail-vroute this is the main thread -- so called "KNI" core, so it is the “vhost0” core and is actually the "lcore-slave-0". we never see this number "0" anywere in any command output. I think we should add this as a "NOTE" or "tip" in the book.

2. why we need 2 lcores for netlink?

3638 4030 ? 00:01:01 lcore-slave-9 #<---service thread
3638 5941 ? 00:00:01 lcore-slave-9 #<---service thread

[kirankn80]

Pl rename KNI core to TAPDEV core. Adding this note will be definitely helpful

For netlink the 2nd thread is a child thread of the first. It is used when qemu is operating in client mode.

[ldurandadomia]

Concerning this remark: regarding this 3638 3638 ? 00:10:07 contrail-vroute this is the main thread -- so called "KNI" core, so it is the “vhost0” core and is actually the "lcore-slave-0". we never see this number "0" anywere in any command output. I think we should add this as a "NOTE" or "tip" in the book.

==> I'm proposing to write: lcore 0: vhost0 lcore0 is indeed the main DPDK vrouter dataplane thread: 3638 3638 ? 00:10:07 contrail-vroute #<---main process

[ldurandadomia]

Nice explanation. I'll update our "perf guide on Google drive" We have to see how we update our book.