search

Mellanox / libvma

Linux user space library for network socket acceleration based on RDMA compatible network adaptors
https://www.mellanox.com/products/software/accelerator-software/vma?mtag=vma
Other
556 stars 152 forks source link

Running vma inside docker container #1058

Open Cha465 opened 6 months ago

Cha465 commented 6 months ago

Hi, is it viable to preload libvma inside docker container, in a loopback scenario (server and client run in the same container)? I preload libvma.so before executable, but the output of strace still shows sendmsg/recvmsg syscalls, and vma_stats finds no offloaded packets.

igor-ivanov commented 6 months ago

Hi @Cha465 the behavior is not container specific. It is expected that scenario using tcp connection on the same server when both (server and client) processes under libvma whould work.

Cha465 commented 6 months ago

Hi, I tried again with sockperf, specifying tcp connection: LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf sr --tcp -i 0.0.0.0 // server VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376 VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13 VMA INFO: Cmd Line: sockperf sr --tcp -i 0.0.0.0 VMA INFO: Current Time: Mon Dec 25 09:31:13 2023 VMA INFO: Pid: 77119 VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0: VMA INFO: Architecture: x86_64 VMA INFO: Node: e55bdebd6779 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA Spec Latency [VMA_SPEC] VMA INFO: Log Level INFO [VMA_TRACELEVEL] VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX] VMA INFO: Tx QP WRE 256 [VMA_TX_WRE] VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING] VMA INFO: Rx QP WRE 256 [VMA_RX_WRE] VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING] VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL] VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL] VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX] VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL] VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE] VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO] VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS] VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL] VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE] VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT] VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC] VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL] VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY] VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD] VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY] VMA INFO: Thread mode Single [VMA_THREAD_MODE] VMA INFO: --------------------------------------------------------------------------- sockperf: == version #3.7-no.git == sockperf: [SERVER] listen on: [ 0] IP = 0.0.0.0 PORT = 11111 # TCP sockperf: Warmup stage (sending a few dummy messages)... sockperf: [tid 77119] using recvfrom() to block on socket(s)

LD_PRELOAD=libvma.so VMA_SPEC=latency sockperf pp -i 0.0.0.0 --tcp -t60 // client VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/rmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/rmem_max, we'll use default 229376 VMA ERROR: utils:424:priv_read_file() ERROR while opening file /proc/sys/net/core/wmem_max (errno 2 No such file or directory) VMA WARNING: utils:443:read_file_to_int() ERROR while getting int from from file /proc/sys/net/core/wmem_max, we'll use default 229376 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA_VERSION: 9.7.1-0 Development Snapshot built on Dec 19 2023 10:37:13 VMA INFO: Cmd Line: sockperf pp -i 0.0.0.0 --tcp -t60 VMA INFO: Current Time: Mon Dec 25 09:31:31 2023 VMA INFO: Pid: 77125 VMA INFO: OFED Version: MLNX_OFED_LINUX-5.4-3.7.5.0: VMA INFO: Architecture: x86_64 VMA INFO: Node: e55bdebd6779 VMA INFO: --------------------------------------------------------------------------- VMA INFO: VMA Spec Latency [VMA_SPEC] VMA INFO: Log Level INFO [VMA_TRACELEVEL] VMA INFO: Ring On Device Memory TX 16384 [VMA_RING_DEV_MEM_TX] VMA INFO: Tx QP WRE 256 [VMA_TX_WRE] VMA INFO: Tx QP WRE Batching 4 [VMA_TX_WRE_BATCHING] VMA INFO: Rx QP WRE 256 [VMA_RX_WRE] VMA INFO: Rx QP WRE Batching 4 [VMA_RX_WRE_BATCHING] VMA INFO: Rx Poll Loops -1 [VMA_RX_POLL] VMA INFO: Rx Prefetch Bytes Before Poll 256 [VMA_RX_PREFETCH_BYTES_BEFORE_POLL] VMA INFO: GRO max streams 0 [VMA_GRO_STREAMS_MAX] VMA INFO: Select Poll (usec) -1 [VMA_SELECT_POLL] VMA INFO: Select Poll OS Force Enabled [VMA_SELECT_POLL_OS_FORCE] VMA INFO: Select Poll OS Ratio 1 [VMA_SELECT_POLL_OS_RATIO] VMA INFO: Select Skip OS 1 [VMA_SELECT_SKIP_OS] VMA INFO: CQ Drain Interval (msec) 100 [VMA_PROGRESS_ENGINE_INTERVAL] VMA INFO: CQ Interrupts Moderation Disabled [VMA_CQ_MODERATION_ENABLE] VMA INFO: CQ AIM Max Count 128 [VMA_CQ_AIM_MAX_COUNT] VMA INFO: CQ Adaptive Moderation Disabled [VMA_CQ_AIM_INTERVAL_MSEC] VMA INFO: CQ Keeps QP Full Disabled [VMA_CQ_KEEP_QP_FULL] VMA INFO: TCP nodelay 1 [VMA_TCP_NODELAY] VMA INFO: Avoid sys-calls on tcp fd Enabled [VMA_AVOID_SYS_CALLS_ON_TCP_FD] VMA INFO: Internal Thread Affinity 0 [VMA_INTERNAL_THREAD_AFFINITY] VMA INFO: Thread mode Single [VMA_THREAD_MODE] VMA INFO: --------------------------------------------------------------------------- sockperf: == version #3.7-no.git == sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s)

[ 0] IP = 0.0.0.0 PORT = 11111 # TCP sockperf : Warmup stage ( sending a few du mmy mess ERSION: 9.7.1-0 Developm ent Snap shot bui sockperf: Test end (interrupted by timer) sockperf: Test ended sockperf: [Total Run] RunTime=60.000 sec; Warm up time=400 msec; SentMessages=4570324; ReceivedMessages=4570323 sockperf: ========= Printing statistics for Server No: 0 sockperf: [Valid Duration] RunTime=59.550 sec; SentMessages=4540388; ReceivedMessages=4540388 sockperf: ====> avg-latency=6.536 (std-dev=2.148) sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0 sockperf: Summary: Latency is 6.536 usec sockperf: Total 4540388 observations; each percentile contains 45403.88 observations sockperf: ---> observation = 2635.175 sockperf: ---> percentile 99.999 = 69.872 sockperf: ---> percentile 99.990 = 17.672 sockperf: ---> percentile 99.900 = 12.896 sockperf: ---> percentile 99.000 = 10.863 sockperf: ---> percentile 90.000 = 6.807 sockperf: ---> percentile 75.000 = 6.566 sockperf: ---> percentile 50.000 = 6.485 sockperf: ---> percentile 25.000 = 6.368 sockperf: ---> observation = 4.769

The performance is not ideal, and vma_stats -p is not showing a single line. What is the cause of this? Maybe the VMA ERROR must be fixed?

igor-ivanov commented 6 months ago

could you try to set specific ip instead of 0.0.0.0.

Cha465 commented 6 months ago

127.0.0.1 or ip inside the docker container give the same result.

igor-ivanov commented 6 months ago

Of course, error should not be appeared. Could you try on server first (w/o container) to separate potential issues.

Example on the same node: Launch server

sudo env VMA_STATS_FILE=/tmp/libvma-sr.log LD_PRELOAD=libvma.so sockperf sr -i 1.1.1.10 --tcp

Statistic for server:

$ cat /tmp/libvma-sr.log
======================================================
        Fd=[24]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
- Foreign Address = [1.1.1.10:50613]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 1 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 2 / 313151 (100.00%) [miss/hit]
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:11111]
Rx poll: 3 / 0 (0.00%) [miss/hit]
======================================================

Launch client:

sudo env VMA_STATS_FILE=/tmp/libvma-cl.log LD_PRELOAD=libvma.so sockperf pp -i 1.1.1.10 --tcp

Statistic for client:

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [1.1.1.10:50613]
- Foreign Address = [1.1.1.10:11111]
Tx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx Offload: 4281 / 313152 / 0 / 0 [kilobytes/packets/eagains/errors]
Rx byte: cur 0 / max 14 / dropped 0 / limit 0
Rx pkt : cur 0 / max 1 / dropped 0
Rx poll: 0 / 313152 (100.00%) [miss/hit]
======================================================
Cha465 commented 6 months ago

I got these results: server

cat /tmp/libvma-sr.log
======================================================
        Fd=[20]
- TCP, Blocked
Rx and Tx where not active
======================================================

client

cat /tmp/libvma-cl.log
======================================================
        Fd=[20]
- TCP, Blocked
- Local Address   = [0.0.0.0:48379]
Rx and Tx where not active
======================================================
igor-ivanov commented 6 months ago

Did you launch on Mellanox device? Did you see any error messages from VMA? Use direct ip as mentioned before.

Cha465 commented 6 months ago
igor-ivanov commented 5 months ago
  • Yes, launched on Mellanox cx6, but I couldn't find a way to manually configure which NIC to use.

Just use ip that is set on Mellanox NIC

  • No error messages except for a warning (I tried ulimit -l unlimited but did not make a difference):
 VMA WARNING: ************************************************************************
 VMA WARNING: Your current max locked memory is: 33762250752. Please change it to unlimited.
 VMA WARNING: Set this user's default to `ulimit -l unlimited`.
 VMA WARNING: Read more about this topic in the VMA's User Manual.
 VMA WARNING: ************************************************************************

It is strange. Did you use different users to set ulimit and launch application?

  • Yes, I used direct ip.