mkstemp(/dev/hugepages/qat/usdm.W7jet6) for hpg_fd failed，SSL_do_handshake() failed

lizj3624 commented 4 years ago

1、version

Deiver： qat1.7.l.4.9.0-00008 Qat_Engine：v0.5.44 OpenSSL-1.1.1g

2、./nginx/sbin/nginx -V

nginx version: openresty/1.15.8.1 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) built with OpenSSL 1.1.1g 21 Apr 2020 TLS SNI support enabled configure arguments: --prefix=/export/servers/OpenResty-1.15.8/nginx --with-cc-opt='-O2 -ggdb -O2 -I/export/servers/OpenResty-1.15.8/include -I/export/servers/OpenSSL-1.1.1g/include -I/export/servers/qat/QATzip/include -I/export/servers/zlib-1.2.11/include -D NGX_SECURE_MEM -D JD_NGX_SSL_HANDSHAKE_TIME -D JD_NGX_HTTP_UPSTREAM_RANDOM -Wno-error=deprecated-declarations' --add-module=../ngx_devel_kit-0.3.1rc1 --add-module=../echo-nginx-module-0.61 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.08 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.15 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.15 --add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.7 --with-ld-opt='-Wl,-rpath,/export/servers/OpenResty-1.15.8/luajit/lib -Wl,-rpath=/export/servers/OpenSSL-1.1.1g/lib -L/export/servers/OpenSSL-1.1.1g/lib -L/export/servers/qat/QATzip/lib64 -lqatzip -L/export/servers/zlib-1.2.11/lib -lz -lssl' --with-pcre=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/pcre-8.39 --with-pcre-jit --with-threads --with-http_auth_request_module --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_slice_module --add-module=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/lua-ssl-nginx-module --add-module=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/ngx_http_dyups_module --add-module=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/ngx_http_sticky_module --with-stream --with-stream_ssl_module --with-openssl-async --with-http_gunzip_module --with-pcre-opt='-g -Ofast -fPIC -m64 -march=native -fstack-protector-strong -D_FORTIFY_SOURCE=2' --add-dynamic-module=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/nginx_qat_module --add-dynamic-module=/root/rpmbuild/BUILD/OpenResty-1.15.8-2.3-56.851dbdb/thirdparty/nginx_qatzip_module --with-stream --with-stream_ssl_preread_module

3、ldd nginx/sbin/nginx

linux-vdso.so.1 =>  (0x00007fff5dadb000)
libqatzip.so.1 => /export/servers/qat/QATzip/lib64/libqatzip.so.1 (0x00007f62f51d1000)
libz.so.1 => /export/servers/zlib-1.2.11/lib/libz.so.1 (0x00007f62f4fb6000)
libssl.so.1.1 => /export/servers/OpenSSL-1.1.1g/lib/libssl.so.1.1 (0x00007f62f4d1f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f62f4b1b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f62f48ff000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f62f46c8000)
libluajit-5.1.so.2 => /export/servers/OpenResty-1.15.8/luajit/lib/libluajit-5.1.so.2 (0x00007f62f4449000)
libm.so.6 => /lib64/libm.so.6 (0x00007f62f4147000)
libcrypto.so.1.1 => /export/servers/OpenSSL-1.1.1g/lib/libcrypto.so.1.1 (0x00007f62f3c8a000)
libc.so.6 => /lib64/libc.so.6 (0x00007f62f38bd000)
libqat_s.so => /export/servers/qat/QAT_Driver/lib/libqat_s.so (0x00007f62f35e8000)
libusdm_drv_s.so => /export/servers/qat/QAT_Driver/lib/libusdm_drv_s.so (0x00007f62f33d0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f62f583c000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f62f31cd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f62f2fb7000)
libudev.so.1 => /lib64/libudev.so.1 (0x00007f62f2da1000)
librt.so.1 => /lib64/librt.so.1 (0x00007f62f2b99000)
libcap.so.2 => /lib64/libcap.so.2 (0x00007f62f2994000)
libdw.so.1 => /lib64/libdw.so.1 (0x00007f62f2745000)
libattr.so.1 => /lib64/libattr.so.1 (0x00007f62f2540000)
libelf.so.1 => /lib64/libelf.so.1 (0x00007f62f2328000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f62f2102000)
libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f62f1ef2000)

4、curl https://my-qat.jd.com:4443/

5、nginx error log:

hugepage_mmap_phy_addr:147 mkstemp(/dev/hugepages/qat/usdm.AmiB7S) for hpg_fd failed with errno: 13 hugepage_alloc_slab:211 mmap on huge page memory allocation failed 2020/06/23 16:17:17 [crit] 10029#10029: *11 SSL_do_handshake() failed (SSL: error:800800DF:lib(128):qat_ecdh_compute_key:presultx pdata malloc failure error:1422F006:SSL routines:ssl_generate_pkey_group:EVP lib) while SSL handshaking, client: 172.28.13.159, server: 172.28.12.190:4443

daweiq commented 4 years ago

Please check the nginx.conf and make sure the account who start worker can access the /dev/hugepages folder.

user root;
worker_processes  1;
#worker_cpu_affinity 1000;
master_process on;
load_module modules/ngx_http_qatzip_filter_module.so;
load_module modules/ngx_ssl_engine_qat_module.so;

lizj3624 commented 4 years ago

add “user root;”，the issues is fixed，thank you

intel / asynch_mode_nginx