multiple device request issue

maryamtahhan commented 1 year ago

The issue is if a single pod requests different devices from different pools it results in multiple uds servers serving the same container and all attempt to mount their uds to the pod as /tmp/afxdp.sock.

A similar issue exists for the AFXDP_DEVICES env var that's set in each container.

This patch fixes the first issue by mounting the xsk socket at /tmp/afxdp_dp//afxdp.sock This patch fixes the second issue by setting the env var in the container to AFXDPDEVICES

[x] We still need to update test code.
[x] Update CNDP
[x] Update DPDK - patch ready to send to DPDK.

maryamtahhan commented 11 months ago

an example of what's set for a pod that requests 2 devices from 2 different pools [access, core] is the following:

[root@dpdk dpdk]# env | grep AFXDP
AFXDP_DEVICES_ACCESS=ens3f0np0
AFXDP_DEVICES_CORE=ens3f1np1

[root@dpdk dpdk]# ls -r /tmp/*

/tmp/ens3f1np1:
afxdp.sock

/tmp/ens3f0np0:
afxdp.sock

maryamtahhan commented 11 months ago

Tested with DPDK

testpmd output

./build/app/dpdk-testpmd --log-level pmd.net.af_xdp:debug  -l 0-2 --no-pci --main-lcore=2 --vdev net_af_xdp0,iface=ens3f0np0,start_queue=22,queue_count=1,use_cni=1,sock=/tmp/afxdp_dp/ens3f0np0/afxdp.sock --vdev net_af_xdp1,iface=ens3f1np1,start_queue=22,queue_count=1,use_cni=1,sock=/tmp/afxdp_dp/ens3f1np1/afxdp.sock -- -i --a --nb-cores=2 --rxq=1 --txq=1 --forward-mode=macswap;
EAL: Detected CPU lcores: 64
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: 8 hugepages of size 1073741824 reserved, but no mounted hugetlbfs found for that size
EAL: VFIO support initialized
rte_pmd_af_xdp_probe(): Initializing pmd_af_xdp for net_af_xdp0
init_internals(): Zero copy between umem and mbuf enabled.
rte_pmd_af_xdp_probe(): Initializing pmd_af_xdp for net_af_xdp1
init_internals(): Zero copy between umem and mbuf enabled.
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Auto-start selected
Set macswap packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
eth_rx_queue_setup(): Set up rx queue, rx queue id: 0, xsk queue id: 22
make_request_cni(): Request: [/connect,dpdk]
make_request_cni(): Response: [/connect,dpdk]
make_request_cni(): Request: [/version]
make_request_cni(): Response: [/version]
make_request_cni(): Request: [/xsk_map_fd,ens3f0np0]
make_request_cni(): Response: [/xsk_map_fd,ens3f0np0]
make_request_cni(): Request: [/fin]
make_request_cni(): Response: [/fin]
Port 0: 40:A6:B7:96:C8:D8
Configuring Port 1 (socket 1)
eth_rx_queue_setup(): Set up rx queue, rx queue id: 0, xsk queue id: 22
make_request_cni(): Request: [/connect,dpdk]
make_request_cni(): Response: [/connect,dpdk]
make_request_cni(): Request: [/version]
make_request_cni(): Response: [/version]
make_request_cni(): Request: [/xsk_map_fd,ens3f1np1]
make_request_cni(): Response: [/xsk_map_fd,ens3f1np1]
make_request_cni(): Request: [/fin]
make_request_cni(): Response: [/fin]

DP logs

INFO[2023-11-27 14:34:54] [udsserver.go:147] [start] Unix domain socket initialised. Listening for new connection.
INFO[2023-11-27 14:35:20] [udsserver.go:162] [start] New connection accepted. Waiting for requests.
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /connect,dpdk
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod unvalidated - Request: /connect,dpdk
DEBU[2023-11-27 14:35:20] [udsserver.go:341] [validatePod] Pod dpdk - Validating pod hostname
DEBU[2023-11-27 14:35:20] [resources_api.go:78] [getPodResources] Opening Pod Resource API connection
DEBU[2023-11-27 14:35:20] [resources_api.go:94] [getPodResources] Requesting pod resource list
DEBU[2023-11-27 14:35:20] [resources_api.go:90] [func2] Closing Pod Resource API connection
DEBU[2023-11-27 14:35:20] [udsserver.go:350] [validatePod] Pod dpdk - Found on node
INFO[2023-11-27 14:35:20] [udsserver.go:382] [validatePod] Pod dpdk is valid for this UDS connection
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: /host_ok
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: /host_ok
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /version
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /version
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: 0.1
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: 0.1
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /xsk_map_fd,ens3f0np0
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /xsk_map_fd,ens3f0np0
DEBU[2023-11-27 14:35:20] [udsserver.go:280] [handleFdRequest] Pod dpdk - Device ens3f0np0 recognised
INFO[2023-11-27 14:35:20] [udsserver.go:261] [writeWithFD] Pod dpdk - Response: /fd_ack, FD: 28
DEBU[2023-11-27 14:35:20] [uds.go:221] [Write] Write: /fd_ack, FD: 28
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /fin
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /fin
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: /fin_ack
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: /fin_ack
DEBU[2023-11-27 14:35:20] [uds.go:298] [cleanup] Closing Unix listener
DEBU[2023-11-27 14:35:20] [uds.go:301] [cleanup] Closing connection
DEBU[2023-11-27 14:35:20] [uds.go:304] [cleanup] Closing socket file
DEBU[2023-11-27 14:35:20] [uds.go:306] [cleanup] Removing socket file
INFO[2023-11-27 14:35:20] [udsserver.go:162] [start] New connection accepted. Waiting for requests.
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /connect,dpdk
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod unvalidated - Request: /connect,dpdk
DEBU[2023-11-27 14:35:20] [udsserver.go:341] [validatePod] Pod dpdk - Validating pod hostname
DEBU[2023-11-27 14:35:20] [resources_api.go:78] [getPodResources] Opening Pod Resource API connection
DEBU[2023-11-27 14:35:20] [resources_api.go:94] [getPodResources] Requesting pod resource list
DEBU[2023-11-27 14:35:20] [resources_api.go:90] [func2] Closing Pod Resource API connection
DEBU[2023-11-27 14:35:20] [udsserver.go:350] [validatePod] Pod dpdk - Found on node
INFO[2023-11-27 14:35:20] [udsserver.go:382] [validatePod] Pod dpdk is valid for this UDS connection
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: /host_ok
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: /host_ok
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /version
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /version
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: 0.1
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: 0.1
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /xsk_map_fd,ens3f1np1
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /xsk_map_fd,ens3f1np1
DEBU[2023-11-27 14:35:20] [udsserver.go:280] [handleFdRequest] Pod dpdk - Device ens3f1np1 recognised
INFO[2023-11-27 14:35:20] [udsserver.go:261] [writeWithFD] Pod dpdk - Response: /fd_ack, FD: 34
DEBU[2023-11-27 14:35:20] [uds.go:221] [Write] Write: /fd_ack, FD: 34
DEBU[2023-11-27 14:35:20] [uds.go:191] [Read] Read: /fin
DEBU[2023-11-27 14:35:20] [uds.go:208] [Read] Request contains no file descriptor
INFO[2023-11-27 14:35:20] [udsserver.go:248] [read] Pod dpdk - Request: /fin
INFO[2023-11-27 14:35:20] [udsserver.go:253] [write] Pod dpdk - Response: /fin_ack
DEBU[2023-11-27 14:35:20] [uds.go:229] [Write] Write: /fin_ack
DEBU[2023-11-27 14:35:20] [uds.go:298] [cleanup] Closing Unix listener
DEBU[2023-11-27 14:35:20] [uds.go:301] [cleanup] Closing connection
DEBU[2023-11-27 14:35:20] [uds.go:304] [cleanup] Closing socket file
DEBU[2023-11-27 14:35:20] [uds.go:306] [cleanup] Removing socket file

maryamtahhan commented 11 months ago

CNDP PR https://github.com/CloudNativeDataPlane/cndp/pull/351

intel / afxdp-plugins-for-kubernetes

multiple device request issue #81