intel / QAT_Engine

Intel QuickAssist Technology( QAT) OpenSSL Engine (an OpenSSL Plug-In Engine) which provides cryptographic acceleration for both hardware and optimized software using Intel QuickAssist Technology enabled Intel platforms. https://developer.intel.com/quickassist
BSD 3-Clause "New" or "Revised" License
392 stars 127 forks source link

How to set QAT to support symmetric encryption and decryption #123

Open luwenpeng opened 4 years ago

luwenpeng commented 4 years ago

I added the debug log in asynch_mode_nginx and found that SSL_read () / SSL_write () never returned SSL_ERROR_WANT_ASYNC.

In order to determine whether SSL_read / SSL_write was actually offloaded to QAT, I performed the following verification:

1.watch -d cat /sys/kernel/debug/qat_c6xx_0000\:1a\:00.0/fw_counters only has the number of bytes when SSL_do_handshrk, and there is no number of bytes when SSL_read / SSL_wirte.

  1. I turned on the QAT debug log (--enable-qat_debug, --with-qat_debug_file=/path/log), only the RSA asymmetric encryption information was seen in the qat debug log, and the symmetric encryption information was not seen.

Will SSL_read () / SSL_write () return SSL_ERROR_WANT_ASYNC, or will the code that handles SSL_read () / SSL_write () return SSL_ERROR_WANT_ASYNC never run?

luwenpeng commented 4 years ago

During QAT configure, I set --enable-qat_small_pkt_offload. I also set ssl_ciphers 'AES128-SHA: AES256-SHA' in nginx.conf. qat_debug_log still cannot see the symmetric encryption information.

./configure \
    --with-qat_dir=/QAT \
    --with-openssl_dir=/openssl \
    --with-openssl_install_dir=/usr/local/ssl \
    --enable-upstream_driver \
    --enable-usdm \
    --enable-qat_rsa \
    --enable-qat_dsa \
    --enable-qat_dh \
    --enable-qat_ecdh \
    --enable-qat_ecdsa \
    --enable-qat_ciphers \
    --enable-qat_prf \
    --enable-qat_small_pkt_offload \
    --enable-qat_debug \
    --with-qat_debug_file=/home/lwp/qat_debug_log

cat nginx.conf

worker_processes  2;

error_log logs/error.log debug;
load_module modules/ngx_ssl_engine_qat_module.so;

events {
}

# Enable QAT engine in heretic mode.
ssl_engine {
    use_engine qat;
    default_algorithms ALL;
    qat_engine {
        qat_offload_mode async;
        qat_notify_mode poll;
        qat_poll_mode heuristic;
    }
}

http {
    gzip off;
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
    }

    # HTTPS server with async mode.
    server {
        #If QAT Engine enabled,  `asynch` need to add to `listen` directive or just add `ssl_asynch  on;` to the context.
        listen       443 ssl asynch;
        server_name  www.test.com;

        ssl_protocols         SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers             AES128-SHA:AES256-SHA;
        ssl_certificate         testg.com-cer.cer;
        ssl_certificate_key  test.com-key.pem;

        location / {
            root   html;
            index  index.html index.htm;
        }
    }
}
paulturx commented 4 years ago

Hi

Testing with nginx will only exercise the QAT in asynchronous mode.
I notice that qat debug is enabled in your configure command which will generate a lot of output.
Also minor point:- --enable-qat_rsa \ --enable-qat_dsa \ --enable-qat_dh \ --enable-qat_ecdh \ --enable-qat_ecdsa \ --enable-qat_ciphers \ --enable-qat_prf \ are all enabled by default so it is not necessary to include them in the configure command. If you let me know versions of all software you are using:- nginx, OpenSSL, QAT engine, QAT driver and the client application you are running to supply traffic to the nginx, I can set up a similar system to help understand your query. Thanks.

paulturx commented 4 years ago

QAT engine does not have to be 'set' as such to operate synchronously. for example, the OpenSSL command:- ./openssl speed -engine qat -elapsed rsa2048

will cause synchronous offload to QAT.

Please refer to the README.md file in the QAT Engine release.

luwenpeng commented 4 years ago

Thank you for your reply, the software version I use is as follows:

openssl version

[root@localhost ~]# /openssl/apps/openssl version
OpenSSL 1.1.1e-dev  xx XXX xxxx

Nginx version

[root@localhost sbin]# ./nginx -V
nginx version: nginx/1.16.1
built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC)
built with OpenSSL 1.1.1e-dev  xx XXX xxxx
TLS SNI support enabled
configure arguments: --prefix=/data/modules/asynch_mode_nginx --with-http_ssl_module --add-dynamic-module=modules/nginx_qat_module/ --with-cc-opt='-DNGX_SECURE_MEM -I /usr/local/ssl/include -Wno-error=deprecated-declarations' --with-ld-opt='-Wl,-rpath=/usr/local/ssl/lib -L /usr/local/ssl/lib'

QAT_driver version

[root@localhost QAT]# cat versionfile
PACKAGE_TYPE=QAT1.7

PACKAGE_OS=L

PACKAGE_VERSION_MAJOR_NUMBER=4

PACKAGE_VERSION_MINOR_NUMBER=7

PACKAGE_VERSION_PATCH_NUMBER=0

PACKAGE_VERSION_BUILD_NUMBER=00006

I don't know how to show QAT_Engine version, it just git clone from master, tag is v0.5.42

client application is curl

[root@localhost home]# curl --version
curl 7.61.1 (x86_64-redhat-linux-gnu) libcurl/7.61.1 OpenSSL/1.1.1 zlib/1.2.11 brotli/1.0.6 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.5/openssl/zlib nghttp2/1.33.0
Release-Date: 2018-09-05
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz brotli TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL Metalink
[root@localhost home]#

In Asynchronous mode, not Synchronous mode:

My doubt is why SSL_read () / SSL_write () does not return SSL_ERROR_WANT_ASYNC?

Qat_debug_log has SSL_do_handshark encryption information. Why is there no encryption information when SSL_read / SSL_write?

paulturx commented 4 years ago

Hi Again I'm not sure I'm understanding completely.

The QAT debug log will only log messages from the QAT Engine, which if running nginx will only show asynchronous calls that are being offloaded to the QAT engine, to the QAT debug file. SSL_read() and SSL_write are OpenSSL calls. Have you tried editing those functions to output debug info and recompiling the OpenSSL. The output won't appear in the QAT debug file but in either the error.log file produced by nginx or else maybe /var/log/messages.

Or else run the speed command I posted yesterday synchronously for a short time (seconds 1) and have a look in the QAT debug log after running for the synchronous info.

luwenpeng commented 4 years ago

Hi paulturx: thank you for your reply, I executed the following command and the result is as follows:

  1. When running "openssl speed -engine qat -elapsed rsa2048" or "openssl speed -engine qat -elapsed -async_jobs 2 rsa2048" or async_mode_nginx call SSL_do_handshark()

qat debug log can find rsa encryption information, indicating that qat supports asymmetric encryption(As we expected, qat debug log saw asymmetric encryption information for ssl_do_handshake)

  1. When running "/openssl/apps/openssl speed -engine qat -elapsed -multi 2 -evp aes-128-cbc-hmac-sha1" or "openssl speed -engine qat -elapsed -async_jobs 128 -multi 2 -evp aes-128-cbc-hmac-sha1"

qat debug log can find ciphers encryption information, indicating that qat supports Symmetric encryption

  1. When async_mode_nginx call SSL_read()/SSL_write()

qat debug log can`t find ciphers encryption information, Implies that symmetric encryption performed by SSL_read / SSL_write does not offload to QAT.


So, my doubt is how to ensure that the symmetric encryption performed by ssl_read / ssl_write can be offloaded to QAT

paulturx commented 4 years ago

Have you tried:- ./openssl s_server -cert /root/paulturx/install/nginx_install_yogaBACKUP/conf/ssl.certs/server-rsa2k.cert.pem -key /root/paulturx/install/nginx_install_yogaBACKUP/conf/ssl.certs/server-rsa2k.key.pem -WWW -accept 4411 -cipher AES128-SHA -tls1_2 -nbio -engine qat -debug -msg

on the server and:-

echo "GET /1kb-file.txt HTTP/1.0" | ./openssl s_client -host localhost -port 4411 -cipher AES128-SHA -tls1_2 -ign_eof -nbio on the client side.

I'll need to check whether SSL_read() and/or SSL_write() are called from this exchange.

luwenpeng commented 4 years ago

Running the following command, no symmetric encryption information related to SSL_read / SSL_write was found in the qat debug log.

Does this mean that asymmetric encrypted data using SSL_do_handshark will be offloaded to QAT? and symmetric encrypted data using SSL_read / SSL_write will not be offloaded to QAT?

./openssl s_server -cert /home/lwp/certs/server.crt -key /home/lwp/certs/server.key -WWW -accept 4411 -cipher AES128-SHA -tls1_2 -nbio -engine qat -debug -msg
echo "GET /1kb-file.txt HTTP/1.0" | ./openssl s_client -host localhost -port 4411 -cipher AES128-SHA -tls1_2 -ign_eof -nbio

Below are the details of the server side / client side / qat side output, respectively.

result on server side

[root@localhost apps]# ./openssl s_server -cert /home/lwp/certs/server.crt -key /home/lwp/certs/server.key -WWW -accept 4411 -cipher AES128-SHA -tls1_2 -nbio -engine qat -debug -msg
engine "qat" set.
Using default temp DH parameters
ACCEPT
Turned on non blocking io
read from 0x26bf700 [0x26cb423] (5 bytes => 5 (0x5))
0000 - 16 03 01 00 71                                    ....q
<<< ??? [length 0005]
    16 03 01 00 71
read from 0x26bf700 [0x26cb428] (113 bytes => 113 (0x71))
0000 - 01 00 00 6d 03 03 93 74-e1 9c 50 8b 78 ed 89 f2   ...m...t..P.x...
0010 - 4c 1e 4e 0a 4d af 37 36-36 a3 e5 0c 86 39 a2 13   L.N.M.766....9..
0020 - 5e 23 82 6a 62 97 00 00-04 00 2f 00 ff 01 00 00   ^#.jb...../.....
0030 - 40 00 23 00 00 00 16 00-00 00 17 00 00 00 0d 00   @.#.............
0040 - 30 00 2e 04 03 05 03 06-03 08 07 08 08 08 09 08   0...............
0050 - 0a 08 0b 08 04 08 05 08-06 04 01 05 01 06 01 03   ................
0060 - 03 02 03 03 01 02 01 03-02 02 02 04 02 05 02 06   ................
0070 - 02                                                .
<<< TLS 1.3, Handshake [length 0071], ClientHello
    01 00 00 6d 03 03 93 74 e1 9c 50 8b 78 ed 89 f2
    4c 1e 4e 0a 4d af 37 36 36 a3 e5 0c 86 39 a2 13
    5e 23 82 6a 62 97 00 00 04 00 2f 00 ff 01 00 00
    40 00 23 00 00 00 16 00 00 00 17 00 00 00 0d 00
    30 00 2e 04 03 05 03 06 03 08 07 08 08 08 09 08
    0a 08 0b 08 04 08 05 08 06 04 01 05 01 06 01 03
    03 02 03 03 01 02 01 03 02 02 02 04 02 05 02 06
    02
>>> ??? [length 0005]
    16 03 03 00 3d
>>> TLS 1.2, Handshake [length 003d], ServerHello
    02 00 00 39 03 03 3b ca 01 8e f2 6d a2 15 3a 3b
    05 dd f6 2b 9b 76 71 20 f6 18 8e a2 98 76 8b 34
    98 02 85 e5 2d 1b 00 00 2f 00 00 11 ff 01 00 01
    00 00 23 00 00 00 16 00 00 00 17 00 00
>>> ??? [length 0005]
    16 03 03 03 e4
>>> TLS 1.2, Handshake [length 03e4], Certificate
    0b 00 03 e0 00 03 dd 00 03 da 30 82 03 d6 30 82
    02 be a0 03 02 01 02 02 01 00 30 0d 06 09 2a 86
    48 86 f7 0d 01 01 0b 05 00 30 81 8b 31 0b 30 09
    06 03 55 04 06 13 02 43 4e 31 10 30 0e 06 03 55
    04 08 0c 07 42 45 49 4a 49 4e 47 31 10 30 0e 06
    03 55 04 07 0c 07 42 45 49 4a 49 4e 47 31 0d 30
    0b 06 03 55 04 0a 0c 04 54 45 53 54 31 0d 30 0b
    06 03 55 04 0b 0c 04 54 45 53 54 31 19 30 17 06
    03 55 04 03 0c 10 77 77 77 2e 6c 75 77 65 6e 70
    65 6e 67 2e 63 6e 31 1f 30 1d 06 09 2a 86 48 86
    f7 0d 01 09 01 16 10 6c 77 70 5f 31 39 39 34 40
    31 36 33 2e 63 6f 6d 30 1e 17 0d 31 39 31 32 31
    38 30 32 34 37 30 31 5a 17 0d 32 30 31 32 31 37
    30 32 34 37 30 31 5a 30 54 31 0b 30 09 06 03 55
    04 06 13 02 43 4e 31 10 30 0e 06 03 55 04 08 0c
    07 42 45 49 4a 49 4e 47 31 0d 30 0b 06 03 55 04
    0a 0c 04 54 45 53 54 31 0d 30 0b 06 03 55 04 0b
    0c 04 54 45 53 54 31 15 30 13 06 03 55 04 03 0c
    0c 77 77 77 2e 74 65 73 74 2e 63 6f 6d 30 82 01
    22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00
    03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 d4 9e
    16 2e 56 a8 5c 79 4d 9b b4 8a 28 a8 2d 49 2b 87
    29 2f fb 67 65 a2 aa d5 83 ec 1b 24 28 b2 99 8a
    9c 1e 3a 3e 90 4f 0a 41 3f 65 58 fb b8 2e 5c e9
    32 e2 6e 8c b0 6b 87 80 19 8f 01 60 fa b8 21 8f
    c4 05 3b f0 e1 bf 36 8b 17 a3 85 74 ed c6 02 0b
    5d 93 15 b2 00 2d 6b 01 72 30 18 64 f2 02 3c a1
    c7 b9 b2 c5 03 21 df 95 26 cc 24 16 8b fb 52 dc
    f8 a8 77 5d f3 11 7d f3 98 8a fe 8b e4 61 0b ce
    b6 07 21 8b c4 6b ac 2a 60 3e 2c be 4f df 37 4a
    fe 21 3e f3 aa 4d 8d 93 e2 56 33 d9 65 bd 12 61
    01 bc 60 9f 72 11 17 4b 35 c2 b1 68 5f 05 02 fb
    fa 5b e7 79 7d 65 d4 9a 06 ba 24 ee da 2e cf 2e
    19 46 11 ba 3c 9a 65 a8 aa 29 f6 37 05 00 26 04
    46 3e cd e7 da f2 a9 dd 9a 98 3b db 54 82 db 5d
    db 82 77 0e 46 d1 ea c4 34 ca 28 ce 99 61 30 a4
    94 9d 7f 65 20 36 f9 bc 05 8c 5f 8e bd 2d 02 03
    01 00 01 a3 7b 30 79 30 09 06 03 55 1d 13 04 02
    30 00 30 2c 06 09 60 86 48 01 86 f8 42 01 0d 04
    1f 16 1d 4f 70 65 6e 53 53 4c 20 47 65 6e 65 72
    61 74 65 64 20 43 65 72 74 69 66 69 63 61 74 65
    30 1d 06 03 55 1d 0e 04 16 04 14 1f f0 4c da 16
    dd e4 8f 1d 03 86 de 79 23 4b d6 b8 5b 3d c1 30
    1f 06 03 55 1d 23 04 18 30 16 80 14 05 33 45 51
    d4 ec fc e9 9d 33 5b 9b 31 5b 9a 61 f6 36 47 d3
    30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00 03
    82 01 01 00 af 3c 8b 6a ca 97 d8 37 da 18 81 8f
    d9 03 f7 92 3b ff bc b2 9c d2 ab 6d 40 3e 69 80
    15 35 99 64 02 31 b1 90 37 a5 df ba 73 34 b4 05
    0a 5c fe ca 7e dd 47 f3 63 cc 5e 14 95 c2 f4 79
    42 72 48 43 37 11 d6 8c 6c 05 a8 f4 dd 20 27 9a
    4c 7e 1a 27 e2 88 fe 7c e9 29 2c b4 b3 81 a0 5b
    2e 04 e9 0a b4 78 59 c8 db cb e4 1b cd e8 76 a6
    f0 f0 83 31 9d 8c 32 1d f9 58 21 b7 35 36 f3 18
    da f0 70 cf 83 55 7d aa 87 92 45 12 a1 ee 3c 82
    96 64 df 45 eb 9e a5 8e 72 58 a5 7a 83 dc ed 41
    6c 2a f4 e8 c7 8d 73 eb 19 7d 45 52 ac 7e c4 2a
    3d 47 12 d2 02 15 50 7e 76 65 d3 b4 d5 1e 15 02
    79 15 46 b2 5f f1 36 38 9b 7e 63 04 16 84 0a f1
    a3 70 3c 66 29 a6 e4 d9 4c 5b 63 cd a4 70 4d e3
    48 73 f9 ee 44 67 89 31 8d 00 b5 2a 27 24 05 d0
    99 0e c3 ee a9 32 c0 e6 23 fe f7 3e c5 6b a7 9d
    48 77 e5 70
>>> ??? [length 0005]
    16 03 03 00 04
>>> TLS 1.2, Handshake [length 0004], ServerHelloDone
    0e 00 00 00
write to 0x26bf700 [0x26d8b90] (1076 bytes => 1076 (0x434))
0000 - 16 03 03 00 3d 02 00 00-39 03 03 3b ca 01 8e f2   ....=...9..;....
0010 - 6d a2 15 3a 3b 05 dd f6-2b 9b 76 71 20 f6 18 8e   m..:;...+.vq ...
0020 - a2 98 76 8b 34 98 02 85-e5 2d 1b 00 00 2f 00 00   ..v.4....-.../..
0030 - 11 ff 01 00 01 00 00 23-00 00 00 16 00 00 00 17   .......#........
0040 - 00 00 16 03 03 03 e4 0b-00 03 e0 00 03 dd 00 03   ................
0050 - da 30 82 03 d6 30 82 02-be a0 03 02 01 02 02 01   .0...0..........
0060 - 00 30 0d 06 09 2a 86 48-86 f7 0d 01 01 0b 05 00   .0...*.H........
0070 - 30 81 8b 31 0b 30 09 06-03 55 04 06 13 02 43 4e   0..1.0...U....CN
0080 - 31 10 30 0e 06 03 55 04-08 0c 07 42 45 49 4a 49   1.0...U....BEIJI
0090 - 4e 47 31 10 30 0e 06 03-55 04 07 0c 07 42 45 49   NG1.0...U....BEI
00a0 - 4a 49 4e 47 31 0d 30 0b-06 03 55 04 0a 0c 04 54   JING1.0...U....T
00b0 - 45 53 54 31 0d 30 0b 06-03 55 04 0b 0c 04 54 45   EST1.0...U....TE
00c0 - 53 54 31 19 30 17 06 03-55 04 03 0c 10 77 77 77   ST1.0...U....www
00d0 - 2e 6c 75 77 65 6e 70 65-6e 67 2e 63 6e 31 1f 30   .luwenpeng.cn1.0
00e0 - 1d 06 09 2a 86 48 86 f7-0d 01 09 01 16 10 6c 77   ...*.H........lw
00f0 - 70 5f 31 39 39 34 40 31-36 33 2e 63 6f 6d 30 1e   p_1994@163.com0.
0100 - 17 0d 31 39 31 32 31 38-30 32 34 37 30 31 5a 17   ..191218024701Z.
0110 - 0d 32 30 31 32 31 37 30-32 34 37 30 31 5a 30 54   .201217024701Z0T
0120 - 31 0b 30 09 06 03 55 04-06 13 02 43 4e 31 10 30   1.0...U....CN1.0
0130 - 0e 06 03 55 04 08 0c 07-42 45 49 4a 49 4e 47 31   ...U....BEIJING1
0140 - 0d 30 0b 06 03 55 04 0a-0c 04 54 45 53 54 31 0d   .0...U....TEST1.
0150 - 30 0b 06 03 55 04 0b 0c-04 54 45 53 54 31 15 30   0...U....TEST1.0
0160 - 13 06 03 55 04 03 0c 0c-77 77 77 2e 74 65 73 74   ...U....www.test
0170 - 2e 63 6f 6d 30 82 01 22-30 0d 06 09 2a 86 48 86   .com0.."0...*.H.
0180 - f7 0d 01 01 01 05 00 03-82 01 0f 00 30 82 01 0a   ............0...
0190 - 02 82 01 01 00 d4 9e 16-2e 56 a8 5c 79 4d 9b b4   .........V.\yM..
01a0 - 8a 28 a8 2d 49 2b 87 29-2f fb 67 65 a2 aa d5 83   .(.-I+.)/.ge....
01b0 - ec 1b 24 28 b2 99 8a 9c-1e 3a 3e 90 4f 0a 41 3f   ..$(.....:>.O.A?
01c0 - 65 58 fb b8 2e 5c e9 32-e2 6e 8c b0 6b 87 80 19   eX...\.2.n..k...
01d0 - 8f 01 60 fa b8 21 8f c4-05 3b f0 e1 bf 36 8b 17   ..`..!...;...6..
01e0 - a3 85 74 ed c6 02 0b 5d-93 15 b2 00 2d 6b 01 72   ..t....]....-k.r
01f0 - 30 18 64 f2 02 3c a1 c7-b9 b2 c5 03 21 df 95 26   0.d..<......!..&
0200 - cc 24 16 8b fb 52 dc f8-a8 77 5d f3 11 7d f3 98   .$...R...w]..}..
0210 - 8a fe 8b e4 61 0b ce b6-07 21 8b c4 6b ac 2a 60   ....a....!..k.*`
0220 - 3e 2c be 4f df 37 4a fe-21 3e f3 aa 4d 8d 93 e2   >,.O.7J.!>..M...
0230 - 56 33 d9 65 bd 12 61 01-bc 60 9f 72 11 17 4b 35   V3.e..a..`.r..K5
0240 - c2 b1 68 5f 05 02 fb fa-5b e7 79 7d 65 d4 9a 06   ..h_....[.y}e...
0250 - ba 24 ee da 2e cf 2e 19-46 11 ba 3c 9a 65 a8 aa   .$......F..<.e..
0260 - 29 f6 37 05 00 26 04 46-3e cd e7 da f2 a9 dd 9a   ).7..&.F>.......
0270 - 98 3b db 54 82 db 5d db-82 77 0e 46 d1 ea c4 34   .;.T..]..w.F...4
0280 - ca 28 ce 99 61 30 a4 94-9d 7f 65 20 36 f9 bc 05   .(..a0....e 6...
0290 - 8c 5f 8e bd 2d 02 03 01-00 01 a3 7b 30 79 30 09   ._..-......{0y0.
02a0 - 06 03 55 1d 13 04 02 30-00 30 2c 06 09 60 86 48   ..U....0.0,..`.H
02b0 - 01 86 f8 42 01 0d 04 1f-16 1d 4f 70 65 6e 53 53   ...B......OpenSS
02c0 - 4c 20 47 65 6e 65 72 61-74 65 64 20 43 65 72 74   L Generated Cert
02d0 - 69 66 69 63 61 74 65 30-1d 06 03 55 1d 0e 04 16   ificate0...U....
02e0 - 04 14 1f f0 4c da 16 dd-e4 8f 1d 03 86 de 79 23   ....L.........y#
02f0 - 4b d6 b8 5b 3d c1 30 1f-06 03 55 1d 23 04 18 30   K..[=.0...U.#..0
0300 - 16 80 14 05 33 45 51 d4-ec fc e9 9d 33 5b 9b 31   ....3EQ.....3[.1
0310 - 5b 9a 61 f6 36 47 d3 30-0d 06 09 2a 86 48 86 f7   [.a.6G.0...*.H..
0320 - 0d 01 01 0b 05 00 03 82-01 01 00 af 3c 8b 6a ca   ............<.j.
0330 - 97 d8 37 da 18 81 8f d9-03 f7 92 3b ff bc b2 9c   ..7........;....
0340 - d2 ab 6d 40 3e 69 80 15-35 99 64 02 31 b1 90 37   ..m@>i..5.d.1..7
0350 - a5 df ba 73 34 b4 05 0a-5c fe ca 7e dd 47 f3 63   ...s4...\..~.G.c
0360 - cc 5e 14 95 c2 f4 79 42-72 48 43 37 11 d6 8c 6c   .^....yBrHC7...l
0370 - 05 a8 f4 dd 20 27 9a 4c-7e 1a 27 e2 88 fe 7c e9   .... '.L~.'...|.
0380 - 29 2c b4 b3 81 a0 5b 2e-04 e9 0a b4 78 59 c8 db   ),....[.....xY..
0390 - cb e4 1b cd e8 76 a6 f0-f0 83 31 9d 8c 32 1d f9   .....v....1..2..
03a0 - 58 21 b7 35 36 f3 18 da-f0 70 cf 83 55 7d aa 87   X!.56....p..U}..
03b0 - 92 45 12 a1 ee 3c 82 96-64 df 45 eb 9e a5 8e 72   .E...<..d.E....r
03c0 - 58 a5 7a 83 dc ed 41 6c-2a f4 e8 c7 8d 73 eb 19   X.z...Al*....s..
03d0 - 7d 45 52 ac 7e c4 2a 3d-47 12 d2 02 15 50 7e 76   }ER.~.*=G....P~v
03e0 - 65 d3 b4 d5 1e 15 02 79-15 46 b2 5f f1 36 38 9b   e......y.F._.68.
03f0 - 7e 63 04 16 84 0a f1 a3-70 3c 66 29 a6 e4 d9 4c   ~c......p<f)...L
0400 - 5b 63 cd a4 70 4d e3 48-73 f9 ee 44 67 89 31 8d   [c..pM.Hs..Dg.1.
0410 - 00 b5 2a 27 24 05 d0 99-0e c3 ee a9 32 c0 e6 23   ..*'$.......2..#
0420 - fe f7 3e c5 6b a7 9d 48-77 e5 70 16 03 03 00 04   ..>.k..Hw.p.....
0430 - 0e 00 00 00                                       ....
read from 0x26bf700 [0x26cb423] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
read R BLOCK
read from 0x26bf700 [0x26cb423] (5 bytes => 5 (0x5))
0000 - 16 03 03 01 06                                    .....
<<< ??? [length 0005]
    16 03 03 01 06
read from 0x26bf700 [0x26cb428] (262 bytes => 262 (0x106))
0000 - 10 00 01 02 01 00 91 63-ee 84 3c 8e 81 ab be 0b   .......c..<.....
0010 - cb e9 38 bc 32 41 ef 9d-92 bd c4 1b 01 41 45 d6   ..8.2A.......AE.
0020 - 83 1b 41 5f 40 ed b8 84-4b 5a 9a 59 ce 54 ad a3   ..A_@...KZ.Y.T..
0030 - e2 8c 05 6e 8a a1 5b dc-1d 7e 58 30 98 13 aa a7   ...n..[..~X0....
0040 - 1f 26 bd cc 46 7a c1 44-a0 a8 d4 08 39 a4 98 9e   .&..Fz.D....9...
0050 - 8a f4 f2 00 09 f4 0a af-e5 45 a3 07 46 ba c9 81   .........E..F...
0060 - 70 2e 14 90 dc ec fc 0a-87 34 4f 76 bf 11 d7 b3   p........4Ov....
0070 - 83 d2 c7 f0 b1 f9 05 a3-76 95 50 70 d7 71 42 75   ........v.Pp.qBu
0080 - 13 06 cc 61 20 8b 9a 03-0d 46 9d 3b b6 2f 65 80   ...a ....F.;./e.
0090 - 2b f8 64 7f 9b fb 75 83-79 68 69 bd 21 2a 87 61   +.d...u.yhi.!*.a
00a0 - 92 22 f6 b5 a0 4a 9b 4e-c7 42 2a 42 df 62 4d cf   ."...J.N.B*B.bM.
00b0 - fa a9 ae 79 57 b7 50 69-18 67 0e fd 23 e3 28 4e   ...yW.Pi.g..#.(N
00c0 - dd 7e d6 2c f9 17 9e 0f-91 1b 22 ed 52 93 c0 1b   .~.,......".R...
00d0 - a7 56 23 61 76 ac 9a 6a-39 0d 35 5e 52 1f b3 c9   .V#av..j9.5^R...
00e0 - 16 f8 81 26 6b 9b 35 bf-76 d6 15 48 1e 1c 86 08   ...&k.5.v..H....
00f0 - 93 eb 3e 43 94 f6 f8 63-3b 66 e7 13 7c 5c 00 f3   ..>C...c;f..|\..
0100 - ec 3e d1 d4 30 18                                 .>..0.
<<< TLS 1.2, Handshake [length 0106], ClientKeyExchange
    10 00 01 02 01 00 91 63 ee 84 3c 8e 81 ab be 0b
    cb e9 38 bc 32 41 ef 9d 92 bd c4 1b 01 41 45 d6
    83 1b 41 5f 40 ed b8 84 4b 5a 9a 59 ce 54 ad a3
    e2 8c 05 6e 8a a1 5b dc 1d 7e 58 30 98 13 aa a7
    1f 26 bd cc 46 7a c1 44 a0 a8 d4 08 39 a4 98 9e
    8a f4 f2 00 09 f4 0a af e5 45 a3 07 46 ba c9 81
    70 2e 14 90 dc ec fc 0a 87 34 4f 76 bf 11 d7 b3
    83 d2 c7 f0 b1 f9 05 a3 76 95 50 70 d7 71 42 75
    13 06 cc 61 20 8b 9a 03 0d 46 9d 3b b6 2f 65 80
    2b f8 64 7f 9b fb 75 83 79 68 69 bd 21 2a 87 61
    92 22 f6 b5 a0 4a 9b 4e c7 42 2a 42 df 62 4d cf
    fa a9 ae 79 57 b7 50 69 18 67 0e fd 23 e3 28 4e
    dd 7e d6 2c f9 17 9e 0f 91 1b 22 ed 52 93 c0 1b
    a7 56 23 61 76 ac 9a 6a 39 0d 35 5e 52 1f b3 c9
    16 f8 81 26 6b 9b 35 bf 76 d6 15 48 1e 1c 86 08
    93 eb 3e 43 94 f6 f8 63 3b 66 e7 13 7c 5c 00 f3
    ec 3e d1 d4 30 18
read from 0x26bf700 [0x26cb423] (5 bytes => 5 (0x5))
0000 - 14 03 03 00 01                                    .....
<<< ??? [length 0005]
    14 03 03 00 01
read from 0x26bf700 [0x26cb428] (1 bytes => 1 (0x1))
0000 - 01                                                .
read from 0x26bf700 [0x26cb423] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 44                                    ....D
<<< ??? [length 0005]
    16 03 03 00 44
read from 0x26bf700 [0x26cb428] (68 bytes => 68 (0x44))
0000 - 95 13 e9 08 25 b9 d5 70-96 b2 61 44 d3 49 91 b3   ....%..p..aD.I..
0010 - 65 b8 05 69 07 f8 6e 4c-cc fd 26 46 7b 63 37 8d   e..i..nL..&F{c7.
0020 - b7 8c 08 81 9a 1b fd a4-09 c7 d9 3c 7f 8e bd b1   ...........<....
0030 - e4 b6 90 93 c6 eb 6b f6-fb 84 15 49 bf 37 80 74   ......k....I.7.t
0040 - 9e a5 4d dc                                       ..M.
<<< TLS 1.2, Handshake [length 0010], Finished
    14 00 00 0c 7f 59 ea dd 6f 88 4f 59 2c 87 48 ec
>>> ??? [length 0005]
    16 03 03 00 aa
>>> TLS 1.2, Handshake [length 00aa], NewSessionTicket
    04 00 00 a6 00 00 1c 20 00 a0 6f 14 66 82 da 9a
    3b b6 ed db 41 16 80 86 8b 20 ed 05 a1 4b 2d 9f
    af 8a 7a 09 15 0b 5a f3 b9 38 d0 ad 90 0f ab 06
    a1 be f8 7d b0 75 a0 c9 11 c9 b1 ae 16 4f 2d d8
    27 d2 11 c9 ca 99 fc 72 0d 49 37 45 39 e9 db 87
    5b d6 20 a5 8f b2 f4 f6 1d dc ec e4 ad 83 22 04
    09 6c cf 9a 4b 26 60 2a cc 65 67 e2 81 58 d5 8a
    c3 e0 01 fa 9e af 16 a6 8e 61 3e 00 8a 9b 08 2e
    c1 96 03 b7 3a 75 f5 5d 4f b1 f7 1d ec 07 71 5f
    f3 98 c2 67 a2 a5 28 43 97 b7 15 90 d1 3c 6d 98
    64 6b d9 8c 70 ab ce 1c ab 76
>>> ??? [length 0005]
    14 03 03 00 01
>>> TLS 1.2, ChangeCipherSpec [length 0001]
    01
>>> ??? [length 0005]
    16 03 03 00 44
>>> TLS 1.2, Handshake [length 0010], Finished
    14 00 00 0c 46 d2 a8 ce 33 4d 87 11 01 a8 d7 bf
write to 0x26bf700 [0x26d8b90] (254 bytes => 254 (0xFE))
0000 - 16 03 03 00 aa 04 00 00-a6 00 00 1c 20 00 a0 6f   ............ ..o
0010 - 14 66 82 da 9a 3b b6 ed-db 41 16 80 86 8b 20 ed   .f...;...A.... .
0020 - 05 a1 4b 2d 9f af 8a 7a-09 15 0b 5a f3 b9 38 d0   ..K-...z...Z..8.
0030 - ad 90 0f ab 06 a1 be f8-7d b0 75 a0 c9 11 c9 b1   ........}.u.....
0040 - ae 16 4f 2d d8 27 d2 11-c9 ca 99 fc 72 0d 49 37   ..O-.'......r.I7
0050 - 45 39 e9 db 87 5b d6 20-a5 8f b2 f4 f6 1d dc ec   E9...[. ........
0060 - e4 ad 83 22 04 09 6c cf-9a 4b 26 60 2a cc 65 67   ..."..l..K&`*.eg
0070 - e2 81 58 d5 8a c3 e0 01-fa 9e af 16 a6 8e 61 3e   ..X...........a>
0080 - 00 8a 9b 08 2e c1 96 03-b7 3a 75 f5 5d 4f b1 f7   .........:u.]O..
0090 - 1d ec 07 71 5f f3 98 c2-67 a2 a5 28 43 97 b7 15   ...q_...g..(C...
00a0 - 90 d1 3c 6d 98 64 6b d9-8c 70 ab ce 1c ab 76 14   ..<m.dk..p....v.
00b0 - 03 03 00 01 01 16 03 03-00 44 ed 56 81 f6 48 4b   .........D.V..HK
00c0 - 5f 9c 75 6d e7 05 16 4e-23 fd de f9 32 2d af 94   _.um...N#...2-..
00d0 - fa 44 58 1d e3 9b 1a c8-5d 13 44 94 e1 1d 83 5e   .DX.....].D....^
00e0 - d1 f6 de a0 35 89 88 1f-9d 7d 00 b2 4e 5d fe 52   ....5....}..N].R
00f0 - 28 f4 6d 83 e9 4f 9d c6-13 2b 2c 2a 47 4f         (.m..O...+,*GO
read from 0x26bf700 [0x26cb423] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
read R BLOCK
read from 0x26bf700 [0x26cb423] (5 bytes => 5 (0x5))
0000 - 17 03 03 00 44                                    ....D
<<< ??? [length 0005]
    17 03 03 00 44
read from 0x26bf700 [0x26cb428] (68 bytes => 68 (0x44))
0000 - 39 01 54 83 91 94 62 c8-31 74 78 ff bb a1 f3 a4   9.T...b.1tx.....
0010 - b6 52 61 c6 87 ea 23 55-93 25 dd c6 e2 65 f0 b9   .Ra...#U.%...e..
0020 - bf c9 15 76 7a 9a 7d 99-9b 05 00 ea cd c1 9a 2a   ...vz.}........*
0030 - 53 80 33 db 6a f7 39 f6-76 60 b4 35 35 c6 79 fe   S.3.j.9.v`.55.y.
0040 - 26 3b 88 4e                                       &;.N
>>> ??? [length 0005]
    17 03 03 01 54
write to 0x26bf700 [0x26d4ad3] (345 bytes => 345 (0x159))
0000 - 17 03 03 01 54 b7 0d 19-db b6 2e 17 b4 55 4a 8d   ....T........UJ.
0010 - 92 1e 6a 53 12 24 99 98-89 7a 27 12 e0 7d 48 63   ..jS.$...z'..}Hc
0020 - 0a 32 ef 05 59 3d 3c db-8e 49 59 92 bd 24 e6 87   .2..Y=<..IY..$..
0030 - 14 90 c9 2c b1 39 94 cb-54 a4 28 6f 61 e7 d5 c9   ...,.9..T.(oa...
0040 - 6f cb c4 27 db 79 34 af-36 16 2e d9 b0 6a 6a a9   o..'.y4.6....jj.
0050 - c1 e6 ea 94 47 43 b7 5e-f5 00 47 b4 f7 44 a8 d0   ....GC.^..G..D..
0060 - 23 64 c5 23 bc 2d 86 01-90 62 c1 92 e3 bf a9 8b   #d.#.-...b......
0070 - 1a 56 38 c9 1b 1a d1 b5-9d 60 aa 64 fa 2c 33 0a   .V8......`.d.,3.
0080 - 41 1a e8 01 d3 92 1b a1-36 75 56 fd 51 26 95 ec   A.......6uV.Q&..
0090 - 5b 34 ee 82 bc 9e d4 94-03 2b f7 46 7a 60 af 7a   [4.......+.Fz`.z
00a0 - 2c 58 9f 9c 43 97 53 d7-fc 1c 72 56 b1 97 12 05   ,X..C.S...rV....
00b0 - b0 37 71 e5 0f 74 c1 3a-31 b6 2e 6d 6e aa 92 e9   .7q..t.:1..mn...
00c0 - 8b c2 ee 11 ab a3 ff 62-81 f2 86 aa 38 f0 1f 06   .......b....8...
00d0 - a0 6e 3c d4 e0 73 d4 9b-22 b6 90 60 6a 1d 30 00   .n<..s.."..`j.0.
00e0 - 2d c8 a9 34 e5 6c 3a 36-2d 77 41 f6 32 2b 87 d3   -..4.l:6-wA.2+..
00f0 - 56 fb c9 ef e5 70 7a 10-48 c3 0a ff 7c 6f c9 3d   V....pz.H...|o.=
0100 - e2 73 82 9d b1 47 44 63-cf 79 46 e3 7e 19 7d 3d   .s...GDc.yF.~.}=
0110 - fc 56 fc 92 14 f9 10 c4-ca fa d3 14 8b f5 8d 44   .V.............D
0120 - 98 c9 a4 4a 5c b5 c7 cd-78 fa 8f ff a7 f1 bf e7   ...J\...x.......
0130 - 83 65 93 10 d4 40 fb 28-66 1f 39 6b 97 a7 45 b4   .e...@.(f.9k..E.
0140 - b0 e4 79 e6 a0 68 03 74-d5 fa b4 0c 84 fc 93 2a   ..y..h.t.......*
0150 - 9e a0 32 7b 70 cd 85 cb-0f                        ..2{p....
^C
[root@localhost apps]#

result on client side

[root@localhost apps]# echo "GET /1kb-file.txt HTTP/1.0" | ./openssl s_client -host localhost -port 4411 -cipher AES128-SHA -tls1_2 -ign_eof -nbio
CONNECTED(00000003)
Turned on non blocking io
write R BLOCK
Can't use SSL_get_servername
depth=0 C = CN, ST = BEIJING, O = TEST, OU = TEST, CN = www.test.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = CN, ST = BEIJING, O = TEST, OU = TEST, CN = www.test.com
verify error:num=21:unable to verify the first certificate
verify return:1
read R BLOCK
read R BLOCK
---
Certificate chain
 0 s:C = CN, ST = BEIJING, O = TEST, OU = TEST, CN = www.test.com
   i:C = CN, ST = BEIJING, L = BEIJING, O = TEST, OU = TEST, CN = www.luwenpeng.cn, emailAddress = lwp_1994@163.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID1jCCAr6gAwIBAgIBADANBgkqhkiG9w0BAQsFADCBizELMAkGA1UEBhMCQ04x
EDAOBgNVBAgMB0JFSUpJTkcxEDAOBgNVBAcMB0JFSUpJTkcxDTALBgNVBAoMBFRF
U1QxDTALBgNVBAsMBFRFU1QxGTAXBgNVBAMMEHd3dy5sdXdlbnBlbmcuY24xHzAd
BgkqhkiG9w0BCQEWEGx3cF8xOTk0QDE2My5jb20wHhcNMTkxMjE4MDI0NzAxWhcN
MjAxMjE3MDI0NzAxWjBUMQswCQYDVQQGEwJDTjEQMA4GA1UECAwHQkVJSklORzEN
MAsGA1UECgwEVEVTVDENMAsGA1UECwwEVEVTVDEVMBMGA1UEAwwMd3d3LnRlc3Qu
Y29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1J4WLlaoXHlNm7SK
KKgtSSuHKS/7Z2WiqtWD7BskKLKZipweOj6QTwpBP2VY+7guXOky4m6MsGuHgBmP
AWD6uCGPxAU78OG/NosXo4V07cYCC12TFbIALWsBcjAYZPICPKHHubLFAyHflSbM
JBaL+1Lc+Kh3XfMRffOYiv6L5GELzrYHIYvEa6wqYD4svk/fN0r+IT7zqk2Nk+JW
M9llvRJhAbxgn3IRF0s1wrFoXwUC+/pb53l9ZdSaBrok7touzy4ZRhG6PJplqKop
9jcFACYERj7N59ryqd2amDvbVILbXduCdw5G0erENMoozplhMKSUnX9lIDb5vAWM
X469LQIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIBDQQfFh1PcGVuU1NM
IEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUH/BM2hbd5I8dA4beeSNL
1rhbPcEwHwYDVR0jBBgwFoAUBTNFUdTs/OmdM1ubMVuaYfY2R9MwDQYJKoZIhvcN
AQELBQADggEBAK88i2rKl9g32hiBj9kD95I7/7yynNKrbUA+aYAVNZlkAjGxkDel
37pzNLQFClz+yn7dR/NjzF4UlcL0eUJySEM3EdaMbAWo9N0gJ5pMfhon4oj+fOkp
LLSzgaBbLgTpCrR4Wcjby+Qbzeh2pvDwgzGdjDId+VghtzU28xja8HDPg1V9qoeS
RRKh7jyClmTfReuepY5yWKV6g9ztQWwq9OjHjXPrGX1FUqx+xCo9RxLSAhVQfnZl
07TVHhUCeRVGsl/xNjibfmMEFoQK8aNwPGYppuTZTFtjzaRwTeNIc/nuRGeJMY0A
tSonJAXQmQ7D7qkywOYj/vc+xWunnUh35XA=
-----END CERTIFICATE-----
subject=C = CN, ST = BEIJING, O = TEST, OU = TEST, CN = www.test.com

issuer=C = CN, ST = BEIJING, L = BEIJING, O = TEST, OU = TEST, CN = www.luwenpeng.cn, emailAddress = lwp_1994@163.com

---
No client certificate CA names sent
---
SSL handshake has read 1330 bytes and written 464 bytes
Verification error: unable to verify the first certificate
---
New, SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES128-SHA
    Session-ID: A27E7AB2C0795B3A02AE36C4043D5277250A3588F8B5DF3622D790E6C825BE34
    Session-ID-ctx:
    Master-Key: 2A0384534A83FA17744557FD2CAA6B333357C3C71E0BDE09D677D1081D220749C530CEBAC51C98140CFCDB418A720F5B
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 6f 14 66 82 da 9a 3b b6-ed db 41 16 80 86 8b 20   o.f...;...A....
    0010 - ed 05 a1 4b 2d 9f af 8a-7a 09 15 0b 5a f3 b9 38   ...K-...z...Z..8
    0020 - d0 ad 90 0f ab 06 a1 be-f8 7d b0 75 a0 c9 11 c9   .........}.u....
    0030 - b1 ae 16 4f 2d d8 27 d2-11 c9 ca 99 fc 72 0d 49   ...O-.'......r.I
    0040 - 37 45 39 e9 db 87 5b d6-20 a5 8f b2 f4 f6 1d dc   7E9...[. .......
    0050 - ec e4 ad 83 22 04 09 6c-cf 9a 4b 26 60 2a cc 65   ...."..l..K&`*.e
    0060 - 67 e2 81 58 d5 8a c3 e0-01 fa 9e af 16 a6 8e 61   g..X...........a
    0070 - 3e 00 8a 9b 08 2e c1 96-03 b7 3a 75 f5 5d 4f b1   >.........:u.]O.
    0080 - f7 1d ec 07 71 5f f3 98-c2 67 a2 a5 28 43 97 b7   ....q_...g..(C..
    0090 - 15 90 d1 3c 6d 98 64 6b-d9 8c 70 ab ce 1c ab 76   ...<m.dk..p....v

    Start Time: 1576637436
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: yes
---
HTTP/1.0 200 ok
Content-type: text/plain

Error opening '1kb-file.txt'
140141777478144:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('1kb-file.txt','r')
140141777478144:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
read:errno=0
[root@localhost apps]#

result in qat debug log

[root@localhost lwp]# cat qat_debug_log
[WARNING][e_qat.c:1222:bind_qat()] QAT Warnings enabled.
[DEBUG][e_qat.c:1223:bind_qat()] QAT Debug enabled.
[DEBUG][e_qat.c:1224:bind_qat()] id=qat
[WARNING][e_qat.c:229:validate_configuration_section_name()] Section name is NULL
[WARNING][e_qat.c:1052:qat_engine_ctrl()] CTRL command not implemented
[DEBUG][e_qat.c:441:qat_engine_init()] QAT Engine initialization:
[DEBUG][e_qat.c:442:qat_engine_init()] - External polling: OFF
[DEBUG][e_qat.c:443:qat_engine_init()] - SW Fallback: OFF
[DEBUG][e_qat.c:444:qat_engine_init()] - Inline polling: OFF
[DEBUG][e_qat.c:445:qat_engine_init()] - Internal poll interval: 10000ns
[DEBUG][e_qat.c:446:qat_engine_init()] - Epoll timeout: 1000ms
[DEBUG][e_qat.c:447:qat_engine_init()] - Event driven polling mode: OFF
[DEBUG][e_qat.c:448:qat_engine_init()] - Instance for thread: OFF
[DEBUG][e_qat.c:449:qat_engine_init()] - Max retry count: 5
[DEBUG][e_qat.c:500:qat_engine_init()] Found 3 Cy instances
[DEBUG][e_qat.c:615:qat_engine_init()] Started Instance No: 0 Located on Device: 0
[DEBUG][e_qat.c:615:qat_engine_init()] Started Instance No: 1 Located on Device: 1
[DEBUG][e_qat.c:615:qat_engine_init()] Started Instance No: 2 Located on Device: 2
[DEBUG][qat_polling.c:177:timer_poll_func()] timer_poll_func started
[DEBUG][qat_polling.c:181:timer_poll_func()] timer_poll_func_thread = 0x7f7548f8f700
[DEBUG][qat_rsa.c:1042:qat_rsa_priv_dec()] - Started.
[DEBUG][qat_rsa.c:403:build_decrypt_op_buf()] - Started
[DEBUG][qat_rsa.c:415:build_decrypt_op_buf()] flen = 256, padding = 3
[DEBUG][qat_rsa.c:533:build_decrypt_op_buf()] - Finished
[DEBUG][qat_rsa.c:248:qat_rsa_decrypt()] - Started
[DEBUG][qat_rsa_crt.c:355:qat_rsa_decrypt_CRT()] - Started
=========================
RSA Decrypt Request: 0x26d9be0
instance_handle = 0x26886f0
op_done = 0x7ffc1a533b20
opData: pRecipientPrivateKey->version = 1
opData: pRecipientPrivateKey->privateKeyRepType = 2
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep1.modulusN.pData: Length 0, Address (nil)
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep1.privateExponentD.pData: Length 0, Address (nil)
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep2.prime1P.pData: Length 128, Address 0x7f7548590400
fb 13 8d df bc f7 96 d0 - aa f2 42 fc 42 83 87 3d
89 59 fa 85 ef a6 b4 cf - eb 45 a0 57 f7 a0 23 62
30 01 d8 ae f5 d4 04 9f - 0b d8 9d 12 ff dd 79 ae
62 cf b5 64 47 11 f5 27 - 21 50 6f 33 7c ce 8f 28
50 50 3c a0 0c be bf f0 - 25 f2 a1 36 23 43 ef b7
01 e6 30 ca 1e a7 f5 54 - 3a ae 27 eb 94 1b 31 d8
56 0f c5 30 c8 82 a6 47 - e0 cf 1a 49 39 5e 49 4f
dd df b0 91 e5 99 c6 fb - a6 23 08 8a 1f 53 72 d3
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep2.prime2Q.pData: Length 128, Address 0x7f7548590800
d8 c9 76 66 1d ef d4 56 - 13 65 e7 30 b3 4a da 51
67 56 64 01 59 9c f7 d4 - ed d6 f3 a6 1f 59 46 02
b0 7e 3d bd fa 30 8b a2 - 91 e8 08 15 4b e4 b2 f0
51 b1 2a b4 e8 9c 18 f0 - 32 0b 91 25 54 a8 bb ae
f3 23 46 78 39 f0 64 90 - 4a 16 37 b9 5e 3e 21 06
d8 88 a4 f2 10 c5 b6 1b - 5d 5c 61 f8 74 05 7a ef
ce 10 15 09 45 31 6d 36 - a4 b6 10 05 ac 29 f5 48
e1 59 11 d1 f6 f5 bc 63 - ed 1f 19 c8 76 57 0f ff
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep2.exponent1Dp.pData: Length 128, Address 0x7f7548590c00
a9 73 a6 11 f5 d0 47 fa - 27 15 80 3e 0e 8a 04 91
74 8c c4 b6 f6 6a 55 a7 - ea 0a c0 cf fd f1 17 50
62 44 2c c0 d0 6a ba 7f - b9 18 05 d2 d7 da db 0f
e5 e2 77 72 e9 40 66 6b - 75 61 7a db 8e 05 3e bb
1f 91 37 89 71 70 9f cb - ed 60 96 8b 1a 3b f4 a2
57 e6 ca d0 ee e5 be 7e - 8d 68 12 cb 3c d7 29 c9
ef 33 8f 8b 80 18 25 2f - 67 bb 7e eb 22 8b cf 00
d7 26 ec 5d 5b 4f 70 eb - 50 52 5e a6 11 c4 59 3f
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep2.exponent2Dq.pData: Length 128, Address 0x7f7548591000
04 d8 94 0d fa 53 57 22 - c9 72 38 b5 cc fc 3a 5f
21 79 3b e9 0f d0 c3 75 - 7e 0f 95 08 bc a0 a9 40
fc 23 06 4c 4d e0 71 ee - a3 66 55 6f ec 6f 63 54
9d ee e0 ee 84 58 a0 f2 - 15 9c 4e 95 50 0c 29 8d
e2 9f 82 3e c3 c4 cc ea - b0 cd 48 57 9b 74 b6 16
4a 16 db e1 11 60 ea 0e - 38 8b 0d 21 99 ae 59 ae
2c a4 0d 64 54 a8 a3 31 - 5a 83 3b 5e 75 17 c7 04
e7 0c af aa 4a e3 f9 5f - 1e 78 d8 81 b3 ab 84 8f
qat_rsa_decrypt_CRT: opData: pRecipientPrivateKey->privateKeyRep2.coefficientQInv.pData: Length 128, Address 0x7f7548591400
8a 26 c3 60 30 08 6c 1b - 1f 51 1d 14 67 8e 18 ca
7d e7 ac 38 f2 7f 8a eb - 19 62 b4 3f ff 76 bc de
6e 51 10 e3 76 ab 92 35 - 37 ea ae df d0 3f 12 64
85 db 74 59 a0 f7 7a 1d - 8d 18 38 18 50 a4 16 4d
4c 1a d8 e3 74 42 c9 c7 - eb 6a 77 f5 37 13 6c 59
99 f0 df 21 42 90 86 75 - 23 58 d1 b7 f5 27 a9 08
47 51 30 7d 18 d5 f1 f7 - 26 12 59 3b 96 22 05 0f
4e 29 16 1a bc 85 65 b1 - b7 51 3c 94 d4 dd 22 8c
qat_rsa_decrypt_CRT: opData: inputData.pData: Length 256, Address 0x7f7548591800
91 63 ee 84 3c 8e 81 ab - be 0b cb e9 38 bc 32 41
ef 9d 92 bd c4 1b 01 41 - 45 d6 83 1b 41 5f 40 ed
b8 84 4b 5a 9a 59 ce 54 - ad a3 e2 8c 05 6e 8a a1
5b dc 1d 7e 58 30 98 13 - aa a7 1f 26 bd cc 46 7a
c1 44 a0 a8 d4 08 39 a4 - 98 9e 8a f4 f2 00 09 f4
0a af e5 45 a3 07 46 ba - c9 81 70 2e 14 90 dc ec
fc 0a 87 34 4f 76 bf 11 - d7 b3 83 d2 c7 f0 b1 f9
05 a3 76 95 50 70 d7 71 - 42 75 13 06 cc 61 20 8b
9a 03 0d 46 9d 3b b6 2f - 65 80 2b f8 64 7f 9b fb
75 83 79 68 69 bd 21 2a - 87 61 92 22 f6 b5 a0 4a
9b 4e c7 42 2a 42 df 62 - 4d cf fa a9 ae 79 57 b7
50 69 18 67 0e fd 23 e3 - 28 4e dd 7e d6 2c f9 17
9e 0f 91 1b 22 ed 52 93 - c0 1b a7 56 23 61 76 ac
9a 6a 39 0d 35 5e 52 1f - b3 c9 16 f8 81 26 6b 9b
35 bf 76 d6 15 48 1e 1c - 86 08 93 eb 3e 43 94 f6
f8 63 3b 66 e7 13 7c 5c - 00 f3 ec 3e d1 d4 30 18
output_buf = 0x26da0d0
=========================
=========================
RSA Decrypt Output: 0x26da0d0
qat_rsa_decrypt_CRT: output_buf: Length 256, Address 0x7f7548591c00
00 02 c0 e0 7e 5f 39 75 - a2 ae f4 38 7e 9f 2f 0e
72 90 e5 33 0d 4d b5 e4 - 2d 28 e3 0c c5 6e e8 23
35 38 06 3b b3 9c 01 04 - d9 a8 fc e7 7d f5 be df
07 c3 92 3d d5 06 3f a6 - 77 13 7e 04 c7 9f 06 28
31 05 3d 6b 4c 85 2a 33 - 9b 9c f2 d1 9f 66 6d 99
e4 19 0c 6f 68 26 d4 8f - 40 61 a3 fa 1e 1f 55 a3
34 3b a1 29 d6 84 2c 5f - a9 58 46 2e 94 2b e3 55
4e a9 03 a9 21 72 0b 2b - d3 86 ed af 6d 72 db 95
c8 d1 67 7a ea fd ec 72 - 99 d4 5f 7d 9e 0c ae 8c
2c 31 6b e3 0b 6d dd 7a - d6 75 2f e6 ba ca 76 d4
05 66 d2 17 bd f2 c0 b5 - 51 44 f0 87 9a f7 49 66
1b 2d c5 d0 c4 dc 52 95 - 2e c7 8b ae ed 0f d7 4c
2e ac a7 1a 76 d6 07 e4 - a3 54 a4 c4 9e 91 7b 00
03 03 71 a6 71 6a 07 d8 - 22 3b 0b a5 22 cf eb 67
ef 6c e4 ac 4f 98 36 5f - 26 61 c7 89 82 ad b4 60
fe a4 fa 49 39 b9 a8 94 - 7c aa 84 7c 2c 81 0c 17
=========================
[DEBUG][qat_rsa_crt.c:525:qat_rsa_decrypt_CRT()] - Finished
[DEBUG][qat_rsa.c:210:rsa_decrypt_op_buf_free()] - Started
[DEBUG][qat_rsa.c:233:rsa_decrypt_op_buf_free()] - Finished
[DEBUG][qat_rsa.c:1160:qat_rsa_priv_dec()] - Finished
[DEBUG][qat_prf.c:525:build_tls_prf_op_data()] Value of label = extended master secret
[DEBUG][qat_prf.c:550:build_tls_prf_op_data()] Using USER_DEFINED label = extended master secret
=========================
PRF Op Data: 0x7ffc1a533ab0
tlsOp: USER_DEFINED:
qat_prf_tls_derive: Secret: Length 48, Address 0x7f7548590400
03 03 71 a6 71 6a 07 d8 - 22 3b 0b a5 22 cf eb 67
ef 6c e4 ac 4f 98 36 5f - 26 61 c7 89 82 ad b4 60
fe a4 fa 49 39 b9 a8 94 - 7c aa 84 7c 2c 81 0c 17
qat_prf_tls_derive: Seed: Length 32, Address 0x7f7548590c00
5f f8 9d 81 69 68 31 d5 - c1 d7 c8 7c 87 87 f9 83
10 1a cd 73 33 da 42 bb - 4e da a0 3c a2 b9 1c 76
qat_prf_tls_derive: User Label: Length 22, Address 0x7f7548590800
65 78 74 65 6e 64 65 64 - 20 6d 61 73 74 65 72 20
73 65 63 72 65 74
=========================
=========================
TLS Keygen Request:
instance_handle = 0x269a940
qat_prf_tls_derive: generated_key->pData: Length 48, Address 0x7f7548591400
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
=========================
[DEBUG][qat_prf.c:724:qat_prf_tls_derive()] Calling cpaCyKeyGenTls2
[DEBUG][qat_callback.c:201:qat_crypto_callbackFn()] status 0 verifyResult 1
=========================
TLS Keygen Output: 0x7f7548591000
qat_prf_tls_derive: generated_key->pData: Length 48, Address 0x7f7548591400
2a 03 84 53 4a 83 fa 17 - 74 45 57 fd 2c aa 6b 33
33 57 c3 c7 1e 0b de 09 - d6 77 d1 08 1d 22 07 49
c5 30 ce ba c5 1c 98 14 - 0c fc db 41 8a 72 0f 5b
=========================
qat_prf_tls_derive: Generated key: Length 48, Address 0x7f7548591400
2a 03 84 53 4a 83 fa 17 - 74 45 57 fd 2c aa 6b 33
33 57 c3 c7 1e 0b de 09 - d6 77 d1 08 1d 22 07 49
c5 30 ce ba c5 1c 98 14 - 0c fc db 41 8a 72 0f 5b
[DEBUG][qat_prf.c:525:build_tls_prf_op_data()] Value of label = key expansion
=========================
PRF Op Data: 0x7ffc1a533bd0
tlsOp: KEY_MATERIAL_DERIVE
qat_prf_tls_derive: Secret: Length 48, Address 0x7f7548590400
2a 03 84 53 4a 83 fa 17 - 74 45 57 fd 2c aa 6b 33
33 57 c3 c7 1e 0b de 09 - d6 77 d1 08 1d 22 07 49
c5 30 ce ba c5 1c 98 14 - 0c fc db 41 8a 72 0f 5b
qat_prf_tls_derive: Seed: Length 64, Address 0x7f7548590c00
3b ca 01 8e f2 6d a2 15 - 3a 3b 05 dd f6 2b 9b 76
71 20 f6 18 8e a2 98 76 - 8b 34 98 02 85 e5 2d 1b
93 74 e1 9c 50 8b 78 ed - 89 f2 4c 1e 4e 0a 4d af
37 36 36 a3 e5 0c 86 39 - a2 13 5e 23 82 6a 62 97
qat_prf_tls_derive: User Label: Length 0, Address (nil)
=========================
=========================
TLS Keygen Request:
instance_handle = 0x26a8d90
qat_prf_tls_derive: generated_key->pData: Length 104, Address 0x7f7548591400
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00
=========================
[DEBUG][qat_prf.c:724:qat_prf_tls_derive()] Calling cpaCyKeyGenTls2
[DEBUG][qat_callback.c:201:qat_crypto_callbackFn()] status 0 verifyResult 1
=========================
TLS Keygen Output: 0x7f7548591000
qat_prf_tls_derive: generated_key->pData: Length 104, Address 0x7f7548591400
a6 4e 93 fc ec 7d e7 d6 - 0f a1 a1 4d f4 58 af 2e
89 78 6d 28 eb fc 00 57 - e5 1e f5 a2 05 58 b4 fe
62 72 81 15 82 05 0f b9 - 45 22 d9 e3 9d eb ac 0e
e7 9a 6d 32 8f 6a 40 87 - 9f f6 3a 95 76 0a 96 e6
49 fc 6b 72 a8 44 32 de - 17 fd 7a 63 13 5a d9 d8
6e 9a 2a 03 c0 2f f2 7c - 1d 1e 0e 7b 39 da 35 48
fb 3e 51 32 1e b5 e4 cc
=========================
qat_prf_tls_derive: Generated key: Length 104, Address 0x7f7548591400
a6 4e 93 fc ec 7d e7 d6 - 0f a1 a1 4d f4 58 af 2e
89 78 6d 28 eb fc 00 57 - e5 1e f5 a2 05 58 b4 fe
62 72 81 15 82 05 0f b9 - 45 22 d9 e3 9d eb ac 0e
e7 9a 6d 32 8f 6a 40 87 - 9f f6 3a 95 76 0a 96 e6
49 fc 6b 72 a8 44 32 de - 17 fd 7a 63 13 5a d9 d8
6e 9a 2a 03 c0 2f f2 7c - 1d 1e 0e 7b 39 da 35 48
fb 3e 51 32 1e b5 e4 cc
[DEBUG][qat_prf.c:525:build_tls_prf_op_data()] Value of label = client finished
=========================
PRF Op Data: 0x7ffc1a533b80
tlsOp: CLIENT_FINISHED_DERIVE
qat_prf_tls_derive: Secret: Length 48, Address 0x7f7548590400
2a 03 84 53 4a 83 fa 17 - 74 45 57 fd 2c aa 6b 33
33 57 c3 c7 1e 0b de 09 - d6 77 d1 08 1d 22 07 49
c5 30 ce ba c5 1c 98 14 - 0c fc db 41 8a 72 0f 5b
qat_prf_tls_derive: Seed: Length 32, Address 0x7f7548590c00
5f f8 9d 81 69 68 31 d5 - c1 d7 c8 7c 87 87 f9 83
10 1a cd 73 33 da 42 bb - 4e da a0 3c a2 b9 1c 76
qat_prf_tls_derive: User Label: Length 0, Address (nil)
=========================
=========================
TLS Keygen Request:
instance_handle = 0x26886f0
qat_prf_tls_derive: generated_key->pData: Length 12, Address 0x7f7548591400
00 00 00 00 00 00 00 00 - 00 00 00 00
=========================
[DEBUG][qat_prf.c:724:qat_prf_tls_derive()] Calling cpaCyKeyGenTls2
[DEBUG][qat_callback.c:201:qat_crypto_callbackFn()] status 0 verifyResult 1
=========================
TLS Keygen Output: 0x7f7548591000
qat_prf_tls_derive: generated_key->pData: Length 12, Address 0x7f7548591400
7f 59 ea dd 6f 88 4f 59 - 2c 87 48 ec
=========================
qat_prf_tls_derive: Generated key: Length 12, Address 0x7f7548591400
7f 59 ea dd 6f 88 4f 59 - 2c 87 48 ec
[DEBUG][qat_prf.c:525:build_tls_prf_op_data()] Value of label = server finished
=========================
PRF Op Data: 0x7ffc1a533bb0
tlsOp: SERVER_FINISHED_DERIVE
qat_prf_tls_derive: Secret: Length 48, Address 0x7f7548590400
2a 03 84 53 4a 83 fa 17 - 74 45 57 fd 2c aa 6b 33
33 57 c3 c7 1e 0b de 09 - d6 77 d1 08 1d 22 07 49
c5 30 ce ba c5 1c 98 14 - 0c fc db 41 8a 72 0f 5b
qat_prf_tls_derive: Seed: Length 32, Address 0x7f7548590c00
2e 5f e0 27 bc 63 fe 04 - 55 0f 98 ec f3 74 c6 9a
11 5e 83 04 19 e2 08 a1 - 95 54 ff 0a f8 2f b0 eb
qat_prf_tls_derive: User Label: Length 0, Address (nil)
=========================
=========================
TLS Keygen Request:
instance_handle = 0x269a940
qat_prf_tls_derive: generated_key->pData: Length 12, Address 0x7f7548591400
00 00 00 00 00 00 00 00 - 00 00 00 00
=========================
[DEBUG][qat_prf.c:724:qat_prf_tls_derive()] Calling cpaCyKeyGenTls2
[DEBUG][qat_callback.c:201:qat_crypto_callbackFn()] status 0 verifyResult 1
=========================
TLS Keygen Output: 0x7f7548591000
qat_prf_tls_derive: generated_key->pData: Length 12, Address 0x7f7548591400
46 d2 a8 ce 33 4d 87 11 - 01 a8 d7 bf
=========================
[root@localhost lwp]#
paulturx commented 4 years ago

Hi I put debug message in SSL_read() and SSL_write and they only print with the s_client command. You can get the s_client to use the QAT engine by adding -engine qat to its invocation. Unless you are using a different CRB for the client vs server or else have 2 separate builds on the same CRB, one for client use and other for server use, then once the server has started using the QAT debug file then it effectively prevents the client from writing to the same file, hence you don't see QAT encryption from SSL_read/write. So either use sparate CRB or do not specify a qat debug file at all & all output from QAT encryption/decryption will go to stderr. You should be able to see data that normally would go to qat debug file in the output from the s_client command.

luwenpeng commented 4 years ago
   --enable-qat_debug
# --with-qat_debug_file=/home/lwp/qat_debug_log
./openssl speed -engine qat -elapsed -async_jobs 128 -multi 2 -evp aes-128-cbc-hmac-sha1
echo "GET /1kb-file.txt HTTP/1.0" | ./openssl s_client -host localhost -port 4411 -cipher AES128-SHA -tls1_2 -ign_eof -nbio -engine qat
./openssl s_server -cert /home/lwp/certs/server.crt -key /home/lwp/certs/server.key -WWW -accept 4411 -cipher AES128-SHA -tls1_2 -nbio  -debug -msg -engine qat

Does this mean that qat supports symmetric encryption, but SSL_read / SSL_write of openssl does not use symmetric encryption for qat, but uses software encryption of openssl?

paulturx commented 4 years ago

Hi After some discussion within the team, we think that the problem may be because you are using an OpenSSL 1.1.0 or newer client and server (you say above that you are using OpenSSL 1.1.1e-dev). This is causing Encrypted Then Mac to get negotiated. This means that the chained cipher is not negotiated and the AES operation and hashing operation are not offloaded individually. Please take a look at the URL https://wiki.openssl.org/index.php/List_of_SSL_OP_Flags and in particular the SSL_OP_NO_ENCRYPT_THEN MAC flag and recompile the OpenSSL for use by both client and server with this flag set. If this is the problem then it would explain why 'speed' runs fine since it just uses the chained ciphers. Please let us know if this solves the issue.

luwenpeng commented 4 years ago

Hi paulturx:

thank you for your help!

I modified the source code of OpenSSL 1.1.1e-dev and added the following code to s_client.c / s_server.c:

SSL_CTX_set_options (ctx, SSL_OP_NO_ENCRYPT_THEN_MAC);

After recompiling openssl, use the previous command to test, you can see the symmetric encryption information in the stderror of the server and client.

luwenpeng commented 4 years ago

I checked some information and found that ENCRYPT_THEN_MAC was introduced to solve security problems. ENCRYPT_THEN_MAC is enabled by default in openssl-1.1.1.

Our tests show that symmetric encryption can be offloaded to qat only when ENCRYPT_THEN_MAC is disabled in openssl-1.1.1.

As a production environment, security should be more important, enabling ENCRYPT_THEN_MAC is better.