search

intel / intel-ipsec-mb

Intel(R) Multi-Buffer Crypto for IPSec
BSD 3-Clause "New" or "Revised" License
288 stars 88 forks source link

Crash seen on VMware with dpdk crypto using ipsec-mb library. #153

Open krishnamurthy-dv opened 1 week ago

krishnamurthy-dv commented 1 week ago

We are using IPSec-mb library version 1.2.0 with dpdk for crypto acceleration. On vmware we are seeing a crash below when it tries to execute avx512 instruction. The mb_mgr shows that the AVX512 is detected by library.(see features below). But the vmware proc/cpuinfo does not show the avx512 instruction set as supported. We saw that X86_FEATURE_XSAVE is not enabled in the VM and avx512 instruction depends on this instruction (from https://github.com/torvalds/linux/blob/master/arch/x86/kernel/cpu/cpuid-deps.c). So our suspect is that the ipsecMB library is not considering this XSAVE before detecting AVX/AVX512 feature. Can you please let us know if this is correct and is there any fix for this issue?

0 0x00007f4983153cf6 in init_mb_mgr_avx512 () from /lib/x86_64-linux-gnu/libIPSec_MB.so.1

(gdb) bt

0 0x00007f4983153cf6 in init_mb_mgr_avx512 () from /lib/x86_64-linux-gnu/libIPSec_MB.so.1

1 0x00007f49831c39f5 in init_mb_mgr_auto () from /lib/x86_64-linux-gnu/libIPSec_MB.so.1

2 0x0000558fafe584aa in ipsec_mb_alloc_mgr_from_memzone (mb_mgr_mz_name=0x7ffe83246860 "IMB_MGR_DEV_0_QP_0", mb_mgr_mz=0x10c1b20d0) at ../drivers/crypto/ipsec_mb/ipsec_mb_ops.c:189

3 ipsec_mb_qp_setup (dev=0x558fbbf15480 , qp_id=, qp_conf=0x7ffe83246930, socket_id=0) at ../drivers/crypto/ipsec_mb/ipsec_mb_ops.c:248

(gdb) p *mb_mgr $6 = { flags = 0, features = 315390,

tkanteck commented 1 week ago

Hi, thanks for the contact.

Yes, ipsec-mb doesn't look into XSAVE property and it analyzes ISA capabilities provided by CPUID instruction. Based on this example: features = 315390 => { AESNI = 1, PCLMULQDQ = 1, CMOV = 1, SSE4.2 =1, AVX = 1, AVX2 = 1, AVX512F = 1, AVX512DQ = 1, AVX512CD = 1, AVX512BW = 1, AVX512VL = 1, SAFE_DATA, SAFE_PARAM, BMI2 = 1}.

Could you run disassem command after the crash?

Note: init_mb_mgr_auto() can be replaced with init_mb_mgr_avx2() as a workaround

krishnamurthy-dv commented 1 week ago

Thanks for your quick response. yes the disassembly showed vmovq 0xa265a(%rip),%xmm2

please let us know if there is any fix planned for this in any of the ipsecmb versions?

tkanteck commented 1 week ago

could you check address in RIP register?

This is not an AVX512 instruction and if crash happens here then issue is not related to XSAVE or CPUID. Is it possible to reproduce the problem on your system with library test/imb-kat or perf/imb-perf tools for example? Could try the latest released version, please?

krishnamurthy-dv commented 1 week ago

Attaching one screenshot of intel Instruction set manual which says that vmovq is detected using AVX/AVX512F cpuid feature flags. Please correct me if I am wrong here.

Screenshot 2024-06-28 at 10 24 51 AM
tkanteck commented 5 days ago

True but only if it is EVEX encoded, as stated on the attached screenshot. I expect vmovq 0xa265a(%rip),%xmm2 is VEX encoded and it only needs AVX (disassem needed to confirm).

What CPU do you use? Does the issue happen in VM and/or host OS? Please provide steps to reproduce? Can you reproduce the problem with basic test app (without DPDK)?

krishnamurthy-dv commented 3 days ago

I guess AVX also depends on XSAVE as per kernel source code. Meanwhile trying to get back the setup to recreate this issue.