search

intel / intel-ipsec-mb

Intel(R) Multi-Buffer Crypto for IPSec
BSD 3-Clause "New" or "Revised" License
288 stars 87 forks source link

Port for FreeBSD #62

Closed Martinfx closed 3 years ago

Martinfx commented 3 years ago

I made port for FreeBSD with patches but have small issue.

this is link for port and patches. https://github.com/Martinfx/FreeBSD-Ports/tree/max-intel-ipsec/intel-ipsec-mb

Detected library version: 0.54.0
Detected hardware features:
    SHANI        : n/a
    AESNI        : OK
    PCLMULQDQ    : OK
    CMOV         : OK
    SSE4.2       : OK
    AVX          : OK
    AVX2         : n/a
    AVX512(SKX)  : n/a
    VAES         : n/a
    VPCLMULQDQ   : n/a
    GFNI         : n/a
Testing SSE interface
Known answer passes
Testing aux funcs
AES-CTR standard test vectors:
...............
AES-CTR standard test vectors:
......
...Pass
PON (AES128-CTR/CRC/BIP) test vectors:
.............
...Pass
AES-GCM standard test vectors:

Program received signal SIGBUS, Bus error.
0x00000000002edb36 in aes_keyexp_128_enc_sse ()
(gdb) bt
#0  0x00000000002edb36 in aes_keyexp_128_enc_sse ()
#1  0x00000000016b51ce in aes_gcm_pre_128_sse ()
#2  0x0000000000238a1a in test_gcm_vectors (vector=0x204be0 <gcm_vectors>, encfn=0x239140 <aes_gcm_enc_128>, decfn=0x239220 <aes_gcm_dec_128>) at gcm_test.c:1541
#3  0x00000000002382d6 in test_gcm_std_vectors () at gcm_test.c:1647
#4  gcm_test (p_mgr=<optimized out>) at gcm_test.c:1732
#5  0x00000000002380a9 in main (argc=<optimized out>, argv=<optimized out>) at main.c:288
(gdb) Quit
(gdb) quit
tkanteck commented 3 years ago

Hello @Martinfx

Many thanks for looking into it. Please do not change LINUX define in FreeBSD because it is spread all over the code and if missed in one place it will create segfaults.

At some stage, we'll rename LINUX define into SYSV or something similar. For now, please follow the compilation steps described here

No code changes should be required.

Thanks Tomasz

Martinfx commented 3 years ago

Hello Tomasz, with your recommendation with LINUX define is bad because i cannot compile this code and have more patches for FreeBSD platform because some api from linux is incompatible. I am going to fix this issue.

tkanteck commented 3 years ago

Could you provide shell output from the compilation to help debug the problem?

Martinfx commented 3 years ago

Log from compilation compile.log Log from test with Debug mode test.log

tkanteck commented 3 years ago

I noticed that the compile.log includes -DFreeBSD changes. Could you try to compile without these patches? pull 0.54 code and compile with gmake ?

Martinfx commented 3 years ago

I am sorry this log from test test.log

tkanteck commented 3 years ago

thanks! @pablodelara it seems cross validation tool compilation on FreeBSD fails because of malloc.h header. I guess it could be fixed with ifdef section for FreeBSD there. Could you have a look into it, please?

tkanteck commented 3 years ago

@Martinfx in the mean time could you replace mallo.h with stdlib.h there and check if it compiles fine? thanks

Martinfx commented 3 years ago 
intel-ipsec-mb (max-intel-freebsd-patches)> gmake SHARED=n DEBUG=y -j4
gmake -C lib
gmake[1]: Entering directory '/intel-ipsec-mb/lib'
gmake[1]: Nothing to be done for 'all'.
gmake[1]: Leaving directory '/intel-ipsec-mb/lib'
gmake -C test
gmake[1]: Entering directory '/intel-ipsec-mb/test'
gmake[1]: Nothing to be done for 'all'.
gmake[1]: Leaving directory '/intel-ipsec-mb/test'
gmake -C perf
gmake[1]: Entering directory '/intel-ipsec-mb/perf'
cc -fPIE -z noexecstack -z relro -z now -pthread -L../lib -g ipsec_perf.o msr.o -lIPSec_MB -o ipsec_perf
ld: error: undefined symbol: memalign
>>> referenced by ipsec_perf.c:1113
>>>               ipsec_perf.o:(init_mem)

ld: error: undefined symbol: memalign
>>> referenced by ipsec_perf.c:1123
>>>               ipsec_perf.o:(init_mem)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[1]: *** [Makefile:68: ipsec_perf] Error 1
tkanteck commented 3 years ago

thanks. we'll look into these issues with compiling perf app

pablodelara commented 3 years ago

Hi @Martinfx . We have pushed some commits to fix FreeBSD/clang compilation. You can see here the 4 commits that are fixing the issues. Let us know if these fix the problem.

Martinfx commented 3 years ago

@pablodelara Hi, i send next patch https://github.com/intel/intel-ipsec-mb/pull/64

Martinfx commented 3 years ago

Thank you for fixes but still library have issue. This is log from test. ipsec_MB_testapp.log The issue is Illegal instruction. I have CPU: Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz (2195.07-MHz K8-class CPU)

AES CBC-S 1-9 test vectors (N jobs = 17):
[1/7] Standard vector key_len:16
[2/7] Standard vector key_len:16
[3/7] Standard vector key_len:16
[4/7] Standard vector key_len:16
[5/7] Standard vector key_len:16
[6/7] Standard vector key_len:16
[7/7] Standard vector key_len:16

...Pass
CHACHA20 standard test vectors (N jobs = 1):
[1/2] Standard vector key_len:32
[2/2] Sta
Program received signal SIGILL, Illegal instruction.
0x00000000004c6752 in skip_7 ()
(gdb)

Second issue is maybe in infinite loop in this ./ipsec_xvalid_test


AVX2 not supported. Disabling AVX2 tests
AVX512 not supported. Disabling AVX512 tests
Not using SHA extensions (shani) for SSE arch

Encrypting with Architecture = SSE
    Decrypting with Architecture = SSE
^C
Program received signal SIGINT, Interrupt.
0x0000000001c65732 in loop16_sse ()
(gdb) quit
A debugging session is active.

    Inferior 1 [process 57162] will be killed.
Martinfx commented 3 years ago

@pablodelara ^

pablodelara commented 3 years ago

Hi @Martinfx . Thanks for the patch. I don't know how I could miss that #if in the previous patches. Apologies for that. We will merge your patch soon.

Martinfx commented 3 years ago

@pablodelara Thank you.

pablodelara commented 3 years ago

@Martinfx could you let me know which full name/email address you use in git, so I can change the author name in our internal repo?

pablodelara commented 3 years ago

Hi @Martinfx . Unfortunately I cannot reproduce any of the issues you are showing. In the first issue, can you see the illegal instruction? You should be able to see it using "disassemble".

Martinfx commented 3 years ago

@pablodelara I am sorry the issue is 

./ipsec_xvalid_test
ld-elf.so.1: /usr/local/lib/libIPSec_MB.so.0: Undefined symbol "_aligned_malloc"
pablodelara commented 3 years ago

right, can that be fixed by replacing "#ifdef LINUX" on line 35 with "#if defined (linux) || defined (FreeBSD)" , as you did in your pull request? I cannot see this on my FreeBSD (same version as yours, 12.1). Also, what about the illegal instruction that you were seeing with the test application? Could you get the instruction from disassemble? Make sure you run the application with --no-avx2 --no-avx512, as your CPU does not support these two instruction sets.

Martinfx commented 3 years ago

@pablodelara you had right

JOB API behavior test:
.....
Invalid JOB MAC arguments test:
....................................................................................................................................................................................
Invalid JOB CIPHER arguments test:
........................................................................................................................................
...Pass
Invalid Direct API arguments test:
...
SAFE_PARAM feature disabled, skipping remaining tests
...Pass
Clear memory API test:
....................................................................................................Pass
Starting CRC Test: CRC32 ETHERNET FCS 0x04c11db7
...Pass
Process 98929 stopped
* thread #1, name = 'ipsec_MB_testapp', stop reason = signal SIGILL: privileged instruction
    frame #0: 0x0000000000363d85 ipsec_MB_testapp`init_mb_mgr_avx2 + 293
ipsec_MB_testapp`init_mb_mgr_avx2:
->  0x363d85 <+293>: vpcmpeqd %ymm0, %ymm0, %ymm0
    0x363d89 <+297>: movq   -0x38(%rbp), %rcx
    0x363d8d <+301>: vmovdqa %ymm0, 0x1180(%rcx)
    0x363d95 <+309>: vxorps %xmm1, %xmm1, %xmm1
pablodelara commented 3 years ago

Right, this happens because your CPU does not have AVX2, so passing --no-avx2 --no-avx512 should solve the problem :)

Martinfx commented 3 years ago

@pablodelara I added next commits to pull request

Martinfx commented 3 years ago

@pablodelara yes, i can disable AVX2 and problem is solved but your library this detection must detect automatic. This is bug. I can send next commit :-) .

pablodelara commented 3 years ago

@Martinfx Actually, this is not a problem of the library, but maybe of the application, as it is the app's responsibility to initialize the manager, calling init_mb_mgr_ARCH. Btw, is it really necessary to change #ifdef LINUX to #ifdefined (linux) || # defined (FreeBSD)? The first one should work... don't you see that happening?

Martinfx commented 3 years ago

@pablodelara sometimes we have problem with Undefined symbol "_aligned_malloc" on FreeBSD from github master. In my opion is __linux__ or __FreeBSD__ is better. Issue is in detect OS.

Martinfx commented 3 years ago

In my opinion is this bug on FreeBSD. I must add detect for FreeBSD OS.

Invalid JOB MAC arguments test:
....................................................................................................................................................................................
Invalid JOB CIPHER arguments test:
........................................................................................................................................
...Pass
Invalid Direct API arguments test:
...
SAFE_PARAM feature disabled, skipping remaining tests
...Pass
Clear memory API test:
....................................................................................................Pass
Starting CRC Test: CRC32 ETHERNET FCS 0x04c11db7
...Pass
Process 79619 stopped
* thread #1, name = 'ipsec_MB_testapp', stop reason = signal SIGILL: privileged instruction
    frame #0: 0x0000000000363d85 ipsec_MB_testapp`init_mb_mgr_avx2 + 293
ipsec_MB_testapp`init_mb_mgr_avx2:
->  0x363d85 <+293>: vpcmpeqd %ymm0, %ymm0, %ymm0
    0x363d89 <+297>: movq   -0x38(%rbp), %rcx
    0x363d8d <+301>: vmovdqa %ymm0, 0x1180(%rcx)
    0x363d95 <+309>: vxorps %xmm1, %xmm1, %xmm1
(lldb) quit
tkanteck commented 3 years ago

What CPU are you running the application on?

tkanteck commented 3 years ago

Can you see the same problem if ipsec_MB_testapp runs with "--auto-detect" option?

Martinfx commented 3 years ago

This is with ./ipsec_MB_testapp --auto-detect test.log

tkanteck commented 3 years ago

Thanks! From the log I see the system you run the app on doesn't support AVX2 and the crash is expected. For legacy reasons and for covering negative scenarios "--auto-detect" is not set by default. We'll see if there is something we can do about it.

Martinfx commented 3 years ago

@tkanteck This detect is right with compile option SHARED=n but with option SHARED=y is still bug in code. 

./ipsec_xvalid_test -v
ld-elf.so.1: /usr/local/lib/libIPSec_MB.so.0: Undefined symbol "_aligned_malloc"
pablodelara commented 3 years ago

@Martinfx could you try again with the latest code from the repo, but making sure you clean the old .so? Also, make sure LINUX is still defined in the Makefile, and not replaced by FreeBSD

pablodelara commented 3 years ago

@Martinfx any update from your side? Can we close this issue?

Martinfx commented 3 years ago

@pablodelara yes, Thank you.

Martinfx commented 3 years ago

Thank you for patches for FreeBSD. This is complete port https://www.freshports.org/security/intel-ipsec-mb/ M.

tkanteck commented 3 years ago

Thanks @Martinfx ! this is great!