search

intel / QAT_Engine

Intel QuickAssist Technology( QAT) OpenSSL Engine (an OpenSSL Plug-In Engine) which provides cryptographic acceleration for both hardware and optimized software using Intel QuickAssist Technology enabled Intel platforms. https://developer.intel.com/quickassist
BSD 3-Clause "New" or "Revised" License
395 stars 127 forks source link

Thermal Stress Application #119

Open ericluolove opened 4 years ago

ericluolove commented 4 years ago

Hi All,

When I running Thermal Stress Application met below error, how to solve this problem? Thanks!

[root@localhost Desktop]# ./run_tests.sh 3 3 3 0 Restarting all devices. Processing /etc/c6xx_dev0.conf Processing /etc/c6xx_dev1.conf Processing /etc/c6xx_dev2.conf qaeMemInit started qaeMemInit started qaeMemInit started ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [root@localhost Desktop]# [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [root@localhost Desktop]# [error] [error] [error] SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

[error] [error] [error] do_userStart() - : Failed to start services

do_userStart() - : Failed to start services

do_userStart() - : Failed to start services

ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. main():711 Could not start sal for user space main():711 Could not start sal for user space main():711 Could not start sal for user space

real 0m6.424s user 0m0.002s sys 0m0.005s

real 0m6.424s user 0m0.001s sys 0m0.006s

real 0m6.424s user 0m0.003s sys 0m0.005s [root@localhost Desktop]#

ericluolove commented 4 years ago

Both RHEL 8.0 and SLES 15 SP1 can reproduced this problem.

jdschuet commented 4 years ago

A few questions:

Thanks, Joel

ericluolove commented 4 years ago

Hi Joel, Thanks for your reply.

Which version of QAT driver is installed? qat1.7.l.4.7.0-00006.tar.gz

Was "make samples-install" also performed with QAT driver? Yes, no problem during install QAT driver

Are you able to run performance sample code (cpa_sample_code) Yes, no problem. Run cpa_sample_code with the sign of life option succesfully. ./cpa_sample_code signOfLife=1

jdschuet commented 4 years ago

You are running with the most current version of the QAT software. For these thermal tests you need to run against older version of the QAT driver. https://01.org/sites/default/files/downloads//qat1.7.l.4.4.0-00023.tar.gz

Let me know if this works for you

ericluolove commented 4 years ago

Hi Joel, Not work, prompt as below.

[root@localhost thermal]# ./run_tests.sh 3 3 3 0 Restarting all devices. Processing /etc/c6xx_dev0.conf Processing /etc/c6xx_dev1.conf Processing /etc/c6xx_dev2.conf qaeMemInit started qaeMemInit started qaeMemInit started [root@localhost thermal]# ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 [error] ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances [error] ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed ioctl_alloc_slab:876 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(6) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [root@localhost thermal]# [error] [error] [error] SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

[error] [error] [error] do_userStart() - : Failed to start services

do_userStart() - : Failed to start services

do_userStart() - : Failed to start services

ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL. main():711 Could not start sal for user space main():711 Could not start sal for user space main():711 Could not start sal for user space

real 0m6.426s user 0m0.002s sys 0m0.005s

real 0m6.426s real 0m6.426s user 0m0.002s user 0m0.004s sys 0m0.006s sys 0m0.004s

jdschuet commented 4 years ago

Sorry to hear this did not work for you. Are you up for live debug session?

ericluolove commented 4 years ago

Hi Joel, We are skip the stress at this phase since Intel internal will investigate it. Thanks so much!

jdschuet commented 4 years ago

Hello - I am likely the person that will get contacted to assist. You are using this tool for thermal measurements - correct? Cheers, Joel

ericluolove commented 4 years ago

Yes, you are right...haha

brkyn commented 4 years ago

Hi Joel , I encountered the same problem when run Haproxy with QAT, the error message is same. Could you please let me know if this issue was fixed, or is there any workaround?

Thanks a lot!

jdschuet commented 4 years ago

Hello, A few questions:

Thanks, Joel

ipandtcp commented 4 years ago

Hello, A few questions:

* Which Distro/kernel version?

* Which version of the QAT Driver is used?

* Does the performance sample code (cpa_sample_code) execute properly?

* Which version of OpenSSL?

* Which version of QATEngine

* Are HugePages being used?

* Are you able to query OpenSSL for details on the QAT Engine ( i.e. openssl engine -t -c -vvvv qat)

Thanks, Joel

Hi Joel, I'm workmate with @brkyn. Thank your quick response

The below text google doc link

The text:

[root@localhost bin]# service qat_service status

Checking status of all devices. There is 1 QAT acceleration device(s) in the system: qat_dev0 - type: dh895xcc, inst_id: 0, node_id: 1, bsf: 0000:82:00.0, #accel: 4 #engines: 8 state: up

Application error output:

ioctl_alloc_slab:917 ioctl call for mem allocation failed, ret = -1 ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0) ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed [error] SalCtrl_ServiceInit() - : Failed to initialise all service instances ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL [error] SalCtrl_ServiceEventStart() - : Private data is NULL ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL [error] SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

[error] do_userStart() - : Failed to start services

ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL.

Server hardware:

CPU: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz Mem: 64G

kernel:

version: Linux kernel 3.10, CentOS Linux release 7.3.1611

huge_page: AnonHugePages: 1214464 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB

qat_driver:

version: QAT1.7.L.4.4.0-00023 ./configure make make install

qat_engine:

version: QAT_Engine-0.5.40

Compile: cd /QAT_Engine/qat_contig_mem make make load make test

./configure --with-qat_dir=/QAT --with-openssl_dir=/root/openssl-1.1.0j --with-openssl_install_dir=/usr/local/ssl/ --enable-qat_for_openssl_110 --enable- upstream_driver --enable-usdm

make openssl:
version: openssl-1.1.0j ./config --prefix=/root/openssl/obj/ enable-ssl3 make depend make

application:

haproxy-1.9.10 user: root config: ssl-engine qat ssl-mode-async

openssl outputs:

[root@localhost bin]# ./openssl engine -t -c -vvvv qat

(qat) Reference implementation of QAT crypto engine [RSA, DSA, DH, AES-128-CBC-HMAC-SHA1, AES-128-CBC-HMAC-SHA256, AES-256-CBC-HMAC-SHA1, AES-256-CBC-HMAC-SHA256, TLS1-PRF] [ available ] ENABLE_EXTERNAL_POLLING: Enables the external polling interface to the engine. (input flags): NO_INPUT POLL: Polls the engine for any completed requests (input flags): NO_INPUT SET_INSTANCE_FOR_THREAD: Set instance to be used by this thread (input flags): NUMERIC GET_NUM_OP_RETRIES: Get number of retries (input flags): NO_INPUT SET_MAX_RETRY_COUNT: Set maximum retry count (input flags): NUMERIC SET_INTERNAL_POLL_INTERVAL: Set internal polling interval (input flags): NUMERIC GET_EXTERNAL_POLLING_FD: Returns non blocking fd for crypto engine (input flags): NO_INPUT ENABLE_EVENT_DRIVEN_POLLING_MODE: Set event driven polling mode (input flags): NO_INPUT GET_NUM_CRYPTO_INSTANCES: Get the number of crypto instances (input flags): NO_INPUT DISABLE_EVENT_DRIVEN_POLLING_MODE: Unset event driven polling mode (input flags): NO_INPUT SET_EPOLL_TIMEOUT: Set epoll_wait timeout (input flags): NUMERIC SET_CRYPTO_SMALL_PACKET_OFFLOAD_THRESHOLD: Set QAT small packet threshold (input flags): STRING ENABLE_INLINE_POLLING: Enables the inline polling mode. (input flags): NO_INPUT ENABLE_HEURISTIC_POLLING: Enable the heuristic polling mode (input flags): NO_INPUT GET_NUM_REQUESTS_IN_FLIGHT: Get the number of in-flight requests (input flags): NUMERIC INIT_ENGINE: Initializes the engine if not already initialized (input flags): NO_INPUT

[root@localhost bin]# ./openssl speed -elapsed rsa2048

You have chosen to measure elapsed time instead of user CPU time. Doing 2048 bit private rsa's for 10s: 11226 2048 bit private RSA's in 10.00s Doing 2048 bit public rsa's for 10s: 369307 2048 bit public RSA's in 10.00s OpenSSL 1.1.0j 20 Nov 2018 built on: reproducible build, date unspecified options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/root/ssling/ssl_obj/ssl\"" -DENGINESDIR="\"/root/ssling/ssl_obj/lib/engines-1.1\"" -Wa,--noexecstack sign verify sign/s verify/s rsa 2048 bits 0.000891s 0.000027s 1122.6 36930.7

[root@localhost bin]# ./openssl speed -engine qat -elapsed rsa2048

engine "qat" set. You have chosen to measure elapsed time instead of user CPU time. Doing 2048 bit private rsa's for 10s: 19582 2048 bit private RSA's in 10.00s Doing 2048 bit public rsa's for 10s: 148874 2048 bit public RSA's in 10.00s OpenSSL 1.1.0j 20 Nov 2018 built on: reproducible build, date unspecified options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/root/ssling/ssl_obj/ssl\"" -DENGINESDIR="\"/root/ssling/ssl_obj/lib/engines-1.1\"" -Wa,--noexecstack sign verify sign/s verify/s rsa 2048 bits 0.000511s 0.000067s 1958.2 14887.4

[root@localhost bin]# ./openssl speed -engine qat -elapsed -async_jobs 72 rsa2048

engine "qat" set. You have chosen to measure elapsed time instead of user CPU time. Doing 2048 bit private rsa's for 10s: 219487 2048 bit private RSA's in 10.01s Doing 2048 bit public rsa's for 10s: 2579935 2048 bit public RSA's in 10.00s OpenSSL 1.1.0j 20 Nov 2018 built on: reproducible build, date unspecified options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) blowfish(ptr) compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/root/ssling/ssl_obj/ssl\"" -DENGINESDIR="\"/root/ssling/ssl_obj/lib/engines-1.1\"" -Wa,--noexecstack sign verify sign/s verify/s rsa 2048 bits 0.000046s 0.000004s 21926.8 257993.5

[root@localhost bin]# cat /sys/kernel/debug/qat_dh895xcc_0000\:82\:00.0/fw_counters

+------------------------------------------------+ | FW Statistics for Qat Device | +------------------------------------------------+ | Firmware Requests [AE 0]: 3495220 | | Firmware Responses[AE 0]: 3495220 | +------------------------------------------------+ | Firmware Requests [AE 1]: 3608692 | | Firmware Responses[AE 1]: 3608692 | +------------------------------------------------+ | Firmware Requests [AE 2]: 3494695 | | Firmware Responses[AE 2]: 3494695 | +------------------------------------------------+ | Firmware Requests [AE 3]: 3608649 | | Firmware Responses[AE 3]: 3608649 | +------------------------------------------------+ | Firmware Requests [AE 4]: 3493801 | | Firmware Responses[AE 4]: 3493801 | +------------------------------------------------+ | Firmware Requests [AE 5]: 3609142 | | Firmware Responses[AE 5]: 3609142 | +------------------------------------------------+ | Firmware Requests [AE 6]: 3494898 | | Firmware Responses[AE 6]: 3494898 | +------------------------------------------------+ | Firmware Requests [AE 7]: 3607669 | | Firmware Responses[AE 7]: 3607669 | +------------------------------------------------+

jdschuet commented 4 years ago

Thanks for providing these details. It appears that QAT Engine is properly configured to work with the QAT hardware. On the load of the usdm_drv.ko, are any parameters used to enable hugepages?

At what point are these errors observed? When haproxy launches?

Thanks, Joel

ipandtcp commented 4 years ago

Yes, when HAProxy launches, especially relaunch HAProxy use soft-restart some situation like the new haproxy processes is started and the old HAProxy processes did not exit yet. And at this situation most times not occur errors, but just most times, some other few times HAProxy start failed(error msg: [ALERT] 065/155327 (18361) : ssl-engine qat: failed to initialize), some other few times HAProxy started, but then show errors as I commented above, then HAProxy SSL function not work, but HAProxy other functions worked.

And we're not config any parameters about hugepages, all is kernel default or QAT default. Thanks, Kevin

jdschuet commented 4 years ago

Cool. This indicates that hugepages are not being used.

It also suggests that we are running out of QAT processes when the soft reset is operated. When soft reset occurs QAT resources are still in use while the outstanding requests are finished and the new QAT resources are required for the ones that are spun up. In order to support this model we need to ensure we have enough QAT resources allocated to handle this. This is done by updating NumProcesses in the QAT configuration file. The value provided here needs to be at least 2x the number of HAproxy worker processes. What is the value that is currently there in QAT configuration file?

ipandtcp commented 4 years ago

We are copied config file from "qat/config/dh895xcc/multi_process_event-driven_optimized" to "/etc/dh895xcc_dev0.conf".

And the part of the configuration content is:

# This flag is to enable SSF features (CNV and BnP)
StorageEnabled = 0

##############################################
# Kernel Instances Section
##############################################
[KERNEL]
NumberCyInstances = 0
NumberDcInstances = 0

##############################################
# User Process Instance Section
##############################################
[SHIM]
NumberCyInstances = 1
NumberDcInstances = 0
NumProcesses = 32
LimitDevAccess = 1

# Crypto - User space
Cy0Name = "UserCY0"
Cy0IsPolled = 2
Cy0CoreAffinity = 0-31

And we configured HAProxy worker processes num to 32 (40 CPU cores), i was tried to change QAT config :
NumProcesses = 64
Cy0Coreaffinity = 0-63 but that failed:

[root@localhost haproxy]# service qat_service restart
Restarting all devices.
Processing /etc/dh895xcc_dev0.conf
Ioctl failed
QAT Error: Failed to load config data to device
Failed to configure qat_dev0

So, I understood why HAProxy start failed, but what about HAProxy start ok but SSL function did not work? We configured HAProxy run as 16 threads in 1 worker process before on our old product version, but the problem occurs too.

ipandtcp commented 4 years ago

And thank you, Joel

paulturx commented 4 years ago

Hi Maybe you could reproduce the problem with Warnings enabled in the QAT Engine via the configure option:-

--enable-qat_warnings and then attach a copy of the output in a comment. Please see the README.md file in the QAT engine package. Dependent on your version of QAT engine you may also be able to specify a log file via the --with-qat_debug_file=... option. Please see whether your version of QAT engine supports this option - it will be mentioned in the README.md file if the case.

Regards, paulturx

jdschuet commented 4 years ago

You are most welcome. What is ServicesEnabled set to in the configuration file?

In order to specify more than 32 processes, the ServicesEnabled field must be set to either sym, asym, or cy. Within your application are you looking to accelerate just the handshake (asym crypto)?

ipandtcp commented 4 years ago

Sorry took too long because this issue is really hard to reproduce. The ServicesEnabled we configured is cy just like qat_engine do, actually we copied qat_engine to we're host. And we are looking to accelerate all but it seem just accelerated handshake. I got i some informations from dmesg when the issue occurred, it may helps or it just is a piece of sh*t useless.

[106587.559127] dh895xcc 0000:0f:00.0: device qat_dev0 is busy
[106587.654231] dh895xcc 0000:0f:00.0: device qat_dev0 is busy
[106587.684474] haproxy: page allocation failure: order:9, mode:0x3040d0
[106587.690926] CPU: 4 PID: 76549 Comm: haproxy Tainted: G           OE  ------------   3.10.0 #1
[106587.699537] Hardware name: To be filled by O.E.M. To be filled by O.E.M./CARLOW, BIOS 4.6.5 05/15/2013
[106587.708922]  00000000003040d0 00000000c9dcede6 ffff880112863b78 ffffffff8168707f
[106587.716472]  ffff880112863c08 ffffffff811870a0 0000000000000000 ffff88021f5d1000
[106587.724039]  0000000000000009 00000000003040d0 ffff880112863c08 00000000c9dcede6
[106587.731604] Call Trace:
[106587.734146]  [<ffffffff8168707f>] dump_stack+0x19/0x1b
[106587.739379]  [<ffffffff811870a0>] warn_alloc_failed+0x110/0x180
[106587.745393]  [<ffffffff81682c37>] __alloc_pages_slowpath+0x6b7/0x725
[106587.751834]  [<ffffffff8118b655>] __alloc_pages_nodemask+0x405/0x420
[106587.758272]  [<ffffffff81683bc6>] kmalloc_large_node+0x60/0x8d
[106587.764192]  [<ffffffff811ddca7>] __kmalloc_node+0x247/0x2b0
[106587.769937]  [<ffffffffa068c032>] mem_ioctl+0x452/0x900 [usdm_drv]
[106587.776204]  [<ffffffff812127a5>] do_vfs_ioctl+0x2d5/0x4b0
[106587.781776]  [<ffffffff81692c21>] ? __do_page_fault+0x171/0x450
[106587.787785]  [<ffffffff81212a21>] SyS_ioctl+0xa1/0xc0
[106587.792919]  [<ffffffff81697749>] system_call_fastpath+0x16/0x1b
[106587.799013] Mem-Info:
[106587.801381] active_anon:204108 inactive_anon:116913 isolated_anon:0
 active_file:739022 inactive_file:728439 isolated_file:0
 unevictable:0 dirty:370 writeback:0 unstable:0
 slab_reclaimable:62003 slab_unreclaimable:23245
 mapped:17208 shmem:31658 pagetables:4677 bounce:0
 free:44370 free_pcp:62 free_cma:0
[106587.835910] Node 0 DMA free:15368kB min:128kB low:160kB high:192kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[106587.877559] lowmem_reserve[]: 0 3393 7861 7861
[106587.882182] Node 0 DMA32 free:57560kB min:29108kB low:36384kB high:43660kB active_anon:372584kB inactive_anon:195776kB active_file:1288264kB inactive_file:1277396kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3555416kB managed:3474852kB mlocked:0kB dirty:284kB writeback:0kB mapped:28308kB shmem:55428kB slab_reclaimable:105440kB slab_unreclaimable:30364kB kernel_stack:3632kB pagetables:6984kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[106587.928859] lowmem_reserve[]: 0 0 4468 4468
[106587.933222] Node 0 Normal free:104196kB min:38344kB low:47928kB high:57516kB active_anon:444076kB inactive_anon:271876kB active_file:1667824kB inactive_file:1636360kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4708352kB managed:4575272kB mlocked:0kB dirty:1196kB writeback:0kB mapped:40524kB shmem:71204kB slab_reclaimable:142572kB slab_unreclaimable:62616kB kernel_stack:5984kB pagetables:11724kB unstable:0kB bounce:0kB free_pcp:984kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[106587.980429] lowmem_reserve[]: 0 0 0 0
[106587.984271] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15368kB
[106587.996647] Node 0 DMA32: 2428*4kB (UEM) 773*8kB (UEM) 377*16kB (UEM) 113*32kB (UEM) 108*64kB (UEM) 79*128kB (UEM) 16*256kB (UEM) 11*512kB (U) 5*1024kB (UM) 1*2048kB (M) 0*4096kB = 59464kB
[106588.013901] Node 0 Normal: 9779*4kB (UEM) 2808*8kB (UEM) 163*16kB (UEM) 66*32kB (UEM) 97*64kB (UEM) 62*128kB (UEM) 24*256kB (UM) 14*512kB (UM) 12*1024kB (UEM) 0*2048kB 0*4096kB = 106044kB
[106588.031045] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[106588.039558] 1502840 total pagecache pages
[106588.043660] 3737 pages in swap cache
[106588.047326] Swap cache stats: add 1746230, delete 1742493, find 2122855/2170171
[106588.054718] Free swap  = 33405276kB
[106588.058298] Total swap = 33554428kB
[106588.061878] 2069940 pages RAM
[106588.064935] 0 pages HighMem/MovableOnly
[106588.068862] 53567 pages reserved
[106588.072183] usdm_drv: userMemAlloc:394 Unable to allocate memory slab or wrong alignment:           (null)
[106588.081915] usdm_drv: dev_mem_alloc:599 userMemAlloc failed
[106589.384172] haproxy[76358]: segfault at b96 ip 0000000000000b96 sp 00007f73b80271f8 error 14 in haproxy[400000+191000]
[106589.396193] dh895xcc 0000:0f:00.0: Process 76349 haproxy exit with orphan rings

And I'm continuing to reproduce this issue, When I can easily reproduce, I will add --enable-qat_warnings and other debug options. Thanks.

ipandtcp commented 4 years ago

Hi Is the info from dmesg I commented above have any helps? Thanks.

jdschuet commented 4 years ago

It appears to be memory allocation issue, followed by ungraceful handling of this error. What are the steps required to reproduce?

ipandtcp commented 4 years ago

I sorry I don't know the steps, either, I just restart HAProxy all the time with ab and wrk make HTTPS requests to keep HAProxy and QAT have works to do, not a huge number of requests but contains GET and POST both, and some time restart qat_service too. Then, maybe it happens twice in one minute, maybe no happen in all a day.

jdschuet commented 4 years ago

I understand the restarting of HAproxy as valid use case.. trying to understand the restarting of qat_service. What is the purpose of the restart of QAT?

ipandtcp commented 4 years ago

When half day has gone and error not happen, I will try anything I can imagine, absolutely include restarting of qat_service.

ipandtcp commented 4 years ago

Hi Joel, I'm still can't solve this problem, please give me some help.

ipandtcp commented 4 years ago

@jdschuet

ipandtcp commented 4 years ago

Or tell me how can we avoid this issue are gratefully too.

jdschuet commented 4 years ago

Looking at the software versions installed, and they are quite old. Can you try with later versions. For QAT, the latest package is L.4.9.0-00008 (https://01.org/sites/default/files/downloads/qat1.7.l.4.9.0-00008.tar_0.gz) The latest QAT Engine package is 0.5.43 - https://github.com/intel/QAT_Engine/releases.

ipandtcp commented 4 years ago

I already tried qat1.7.l.4.8.0-00005 and qat_engine0.5.43 with openss1.1.1e, they were the lastest version a few days ago, and the issue still occurs. So I don't think change QAT driver to 4.9 will fix this issue.

jdschuet commented 4 years ago

I agree that moving from 4.8 to 4.9 will likely not resolve the issue. There were memory related fixes that were available in the QAT driver and QAT Engine that were newer that what was reported earlier in this thread.

Are you making using of hugepages? If so how are they configured?

Can you provide details of /proc/meminfo... or better yet monitor the memory when recreating this failure are report results?

ipandtcp commented 4 years ago

I added vm.nr_hugepages=2048 to /etc/sysctl.conf and sysctl show below:

[root@localhost haproxy]# sysctl -a | grep huge
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 2048
vm.nr_hugepages_mempolicy = 2048
vm.nr_overcommit_hugepages = 0
[root@localhost haproxy]# cat /proc/meminfo 
MemTotal:        8065500 kB
MemFree:         1276900 kB
MemAvailable:    2077860 kB
Buffers:             996 kB
Cached:          1124472 kB
SwapCached:            0 kB
Active:          1249948 kB
Inactive:         767076 kB
Active(anon):     892600 kB
Inactive(anon):   128688 kB
Active(file):     357348 kB
Inactive(file):   638388 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554428 kB
SwapFree:       33554428 kB
Dirty:              1532 kB
Writeback:             0 kB
AnonPages:        891788 kB
Mapped:            67208 kB
Shmem:            129728 kB
Slab:              85064 kB
SReclaimable:      48224 kB
SUnreclaim:        36840 kB
KernelStack:        7488 kB
PageTables:        21192 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    35490024 kB
Committed_AS:    6690520 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      385808 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:    491520 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      114776 kB
DirectMap2M:     8165376 kB

And I configured /etc/security/limits.conf as below:

*       hard    nofile      65536
speng soft nofile 10240
speng hard nofile 10240

* hard core 0
* soft  memlock 6710886
* hard memlock  6710886

[root@localhost haproxy]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30691
max locked memory       (kbytes, -l) 6710886
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30691
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

And free:

[root@localhost haproxy]# free -h
              total        used        free      shared  buff/cache   available
Mem:           7.7G        5.3G        1.3G        129M        1.1G        2.0G
Swap:           31G        176K         31G

By the way, I change the platform to Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz and 8G mem, I think this change nothing relate with this issue.

And I will try to capture the meminfos when the problem happening.

Think you, Joel

ipandtcp commented 4 years ago

Hi Joel I catch the meminfos when issue happening

Before restart HAProxy:

free -h
Virtual address ffff880127000000, Physical Address 127000000, Size 2097152
              total        used        free      shared  buff/cache   available
Mem:           7.7G        5.4G        595M        184M        1.7G        1.7G
Swap:           31G        193M         31G

cat /sys/kernel/debug/qae_mem_dbg/qae_mem_slabs
Pid 276405, Slab Id 4297064448
Virtual address ffff880100200000, Physical Address 100200000, Size 2097152
Pid 276405, Slab Id 4318035968
Virtual address ffff880101600000, Physical Address 101600000, Size 2097152
Pid 276405, Slab Id 4326424576
Virtual address ffff880101e00000, Physical Address 101e00000, Size 2097152
Pid 276405, Slab Id 171966464
Virtual address ffff88000a400000, Physical Address a400000, Size 2097152
Pid 276538, Slab Id 4596957184
Virtual address ffff880112000000, Physical Address 112000000, Size 2097152
Pid 276538, Slab Id 4607442944
Virtual address ffff880112a00000, Physical Address 112a00000, Size 2097152
Pid 276538, Slab Id 4592762880
Virtual address ffff880111c00000, Physical Address 111c00000, Size 2097152
Pid 276538, Slab Id 4649385984
Virtual address ffff880115200000, Physical Address 115200000, Size 2097152
Pid 280096, Slab Id 5112856576
Virtual address ffff880130c00000, Physical Address 130c00000, Size 2097152
Pid 280096, Slab Id 4443865088
Virtual address ffff880108e00000, Physical Address 108e00000, Size 2097152
Pid 280096, Slab Id 4445962240
Virtual address ffff880109000000, Physical Address 109000000, Size 2097152
Pid 280096, Slab Id 4448059392
Virtual address ffff880109200000, Physical Address 109200000, Size 2097152
Pid 280215, Slab Id 471859200
Virtual address ffff88001c200000, Physical Address 1c200000, Size 2097152
Pid 280215, Slab Id 4414504960
Virtual address ffff880107200000, Physical Address 107200000, Size 2097152
Pid 280215, Slab Id 4330618880
Virtual address ffff880102200000, Physical Address 102200000, Size 2097152
Pid 280215, Slab Id 4913627136
Virtual address ffff880124e00000, Physical Address 124e00000, Size 2097152
Pid 280216, Slab Id 4383047680
Virtual address ffff880105400000, Physical Address 105400000, Size 2097152
Pid 280216, Slab Id 4328521728
Virtual address ffff880102000000, Physical Address 102000000, Size 2097152
Pid 280216, Slab Id 50331648
Virtual address ffff880003000000, Physical Address 3000000, Size 2097152
Pid 280216, Slab Id 4775215104
Virtual address ffff88011ca00000, Physical Address 11ca00000, Size 2097152
Pid 280217, Slab Id 4366270464
Virtual address ffff880104400000, Physical Address 104400000, Size 2097152
Pid 280217, Slab Id 576716800
Virtual address ffff880022600000, Physical Address 22600000, Size 2097152
Pid 280217, Slab Id 4773117952
Virtual address ffff88011c800000, Physical Address 11c800000, Size 2097152
Pid 280217, Slab Id 4911529984
Virtual address ffff880124c00000, Physical Address 124c00000, Size 2097152
Pid 280218, Slab Id 396361728
Virtual address ffff880017a00000, Physical Address 17a00000, Size 2097152
Pid 280218, Slab Id 5033164800
Virtual address ffff88012c000000, Physical Address 12c000000, Size 2097152
Pid 280218, Slab Id 377487360
Virtual address ffff880016800000, Physical Address 16800000, Size 2097152
Pid 280218, Slab Id 4347396096
Virtual address ffff880103200000, Physical Address 103200000, Size 2097152
Pid 280219, Slab Id 5035261952
Virtual address ffff88012c200000, Physical Address 12c200000, Size 2097152
Pid 280219, Slab Id 520093696
Virtual address ffff88001f000000, Physical Address 1f000000, Size 2097152
Pid 280219, Slab Id 85983232
Virtual address ffff880005200000, Physical Address 5200000, Size 2097152
Pid 280219, Slab Id 4305453056
Virtual address ffff880100a00000, Physical Address 100a00000, Size 2097152
Pid 280221, Slab Id 94371840
Virtual address ffff880005a00000, Physical Address 5a00000, Size 2097152
Pid 280221, Slab Id 4345298944
Virtual address ffff880103000000, Physical Address 103000000, Size 2097152
Pid 280221, Slab Id 4655677440
Virtual address ffff880115800000, Physical Address 115800000, Size 2097152
Pid 280221, Slab Id 255852544
Virtual address ffff88000f400000, Physical Address f400000, Size 2097152
Pid 280220, Slab Id 83886080
Virtual address ffff880005000000, Physical Address 5000000, Size 2097152
Pid 280220, Slab Id 4657774592
Virtual address ffff880115a00000, Physical Address 115a00000, Size 2097152
Pid 280220, Slab Id 4303355904
Virtual address ffff880100800000, Physical Address 100800000, Size 2097152
Pid 280220, Slab Id 257949696
Virtual address ffff88000f600000, Physical Address f600000, Size 2097152
Pid 280223, Slab Id 146800640
Virtual address ffff880008c00000, Physical Address 8c00000, Size 2097152
Pid 280223, Slab Id 4758437888
Virtual address ffff88011ba00000, Physical Address 11ba00000, Size 2097152
Pid 280223, Slab Id 4339007488
Virtual address ffff880102a00000, Physical Address 102a00000, Size 2097152
Pid 280223, Slab Id 4611637248
Virtual address ffff880112e00000, Physical Address 112e00000, Size 2097152
Pid 280226, Slab Id 4756340736
Virtual address ffff88011b800000, Physical Address 11b800000, Size 2097152
Pid 280226, Slab Id 4336910336
Virtual address ffff880102800000, Physical Address 102800000, Size 2097152
Pid 280226, Slab Id 226492416
Virtual address ffff88000d800000, Physical Address d800000, Size 2097152
Pid 280226, Slab Id 5070913536
Virtual address ffff88012e400000, Physical Address 12e400000, Size 2097152
Pid 280227, Slab Id 4450156544
Virtual address ffff880109400000, Physical Address 109400000, Size 2097152
Pid 280227, Slab Id 88080384
Virtual address ffff880005400000, Physical Address 5400000, Size 2097152
Pid 280227, Slab Id 4609540096
Virtual address ffff880112c00000, Physical Address 112c00000, Size 2097152
Pid 280227, Slab Id 4364173312
Virtual address ffff880104200000, Physical Address 104200000, Size 2097152
Pid 280224, Slab Id 4452253696
Virtual address ffff880109600000, Physical Address 109600000, Size 2097152
Pid 280224, Slab Id 148897792
Virtual address ffff880008e00000, Physical Address 8e00000, Size 2097152
Pid 280224, Slab Id 90177536
Virtual address ffff880005600000, Physical Address 5600000, Size 2097152
Pid 280224, Slab Id 461373440
Virtual address ffff88001b800000, Physical Address 1b800000, Size 2097152
Pid 280229, Slab Id 228589568
Virtual address ffff88000da00000, Physical Address da00000, Size 2097152
Pid 280229, Slab Id 5073010688
Virtual address ffff88012e600000, Physical Address 12e600000, Size 2097152
Pid 280229, Slab Id 364904448
Virtual address ffff880015c00000, Physical Address 15c00000, Size 2097152
Pid 280229, Slab Id 4424990720
Virtual address ffff880107c00000, Physical Address 107c00000, Size 2097152
Pid 280228, Slab Id 4731174912
Virtual address ffff88011a000000, Physical Address 11a000000, Size 2097152
Pid 280228, Slab Id 367001600
Virtual address ffff880015e00000, Physical Address 15e00000, Size 2097152
Pid 280228, Slab Id 4357881856
Virtual address ffff880103c00000, Physical Address 103c00000, Size 2097152
Pid 280228, Slab Id 4691329024
Virtual address ffff880117a00000, Physical Address 117a00000, Size 2097152
Pid 280225, Slab Id 463470592
Virtual address ffff88001ba00000, Physical Address 1ba00000, Size 2097152
Pid 280225, Slab Id 4733272064
Virtual address ffff88011a200000, Physical Address 11a200000, Size 2097152
Pid 280225, Slab Id 4359979008
Virtual address ffff880103e00000, Physical Address 103e00000, Size 2097152
Pid 280225, Slab Id 4590665728
Virtual address ffff880111a00000, Physical Address 111a00000, Size 2097152
Pid 280230, Slab Id 4427087872
Virtual address ffff880107e00000, Physical Address 107e00000, Size 2097152
Pid 280230, Slab Id 4397727744
Virtual address ffff880106200000, Physical Address 106200000, Size 2097152
Pid 280230, Slab Id 251658240
Virtual address ffff88000f000000, Physical Address f000000, Size 2097152
Pid 280230, Slab Id 580911104
Virtual address ffff880022a00000, Physical Address 22a00000, Size 2097152
Pid 280222, Slab Id 4487905280
Virtual address ffff88010b800000, Physical Address 10b800000, Size 2097152
Pid 280222, Slab Id 253755392
Virtual address ffff88000f200000, Physical Address f200000, Size 2097152
Pid 280222, Slab Id 578813952
Virtual address ffff880022800000, Physical Address 22800000, Size 2097152
Pid 280222, Slab Id 4949278720
Virtual address ffff880127000000, Physical Address 127000000, Size 2097152

cat /proc/meminfo
MemTotal:        8065500 kB
MemFree:          610116 kB
MemAvailable:    1828736 kB
Buffers:              16 kB
Cached:          1552632 kB
SwapCached:        22280 kB
Active:          1122316 kB
Inactive:        1437704 kB
Active(anon):     440992 kB
Inactive(anon):   755420 kB
Active(file):     681324 kB
Inactive(file):   682284 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554428 kB
SwapFree:       33356696 kB
Dirty:              2784 kB
Writeback:             0 kB
AnonPages:        987580 kB
Mapped:            59672 kB
Shmem:            188604 kB
Slab:             190800 kB
SReclaimable:     147800 kB
SUnreclaim:        43000 kB
KernelStack:        7600 kB
PageTables:        23256 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    35490024 kB
Committed_AS:    7298596 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      395168 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:    520192 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      120920 kB
DirectMap2M:     8159232 kB

Other services restarting

Restarting HAProxy

ioctl_alloc_slab:969 ioctl call for mem allocation failed, ret = -1
ADF_UIO_PROXY err: adf_init_ring: unable to get ringbuf(v:(nil),p:(nil)) for rings in bank(0)
ADF_UIO_PROXY err: icp_adf_transCreateHandle: adf_init_ring failed
[error] SalCtrl_ServiceInit() - : Failed to initialise all service instances
ADF_UIO_PROXY err: adf_user_subsystemInit: Failed to initialise Subservice SAL
[error] SalCtrl_ServiceEventStart() - : Private data is NULL
ADF_UIO_PROXY err: adf_user_subsystemStart: Failed to start Subservice SAL

cat /sys/kernel/debug/qae_mem_dbg/qae_mem_slabs
Pid 276405, Slab Id 4297064448
Virtual address ffff880100200000, Physical Address 100200000, Size 2097152
Pid 276405, Slab Id 4318035968
Virtual address ffff880101600000, Physical Address 101600000, Size 2097152
Pid 276405, Slab Id 4326424576
Virtual address ffff880101e00000, Physical Address 101e00000, Size 2097152
Pid 276405, Slab Id 171966464
Virtual address ffff88000a400000, Physical Address a400000, Size 2097152
Pid 276538, Slab Id 4596957184
Virtual address ffff880112000000, Physical Address 112000000, Size 2097152
Pid 276538, Slab Id 4607442944
Virtual address ffff880112a00000, Physical Address 112a00000, Size 2097152
Pid 276538, Slab Id 4592762880
Virtual address ffff880111c00000, Physical Address 111c00000, Size 2097152
Pid 276538, Slab Id 4649385984
Virtual address ffff880115200000, Physical Address 115200000, Size 2097152
Pid 280096, Slab Id 5112856576
Virtual address ffff880130c00000, Physical Address 130c00000, Size 2097152
Pid 280096, Slab Id 4443865088
Virtual address ffff880108e00000, Physical Address 108e00000, Size 2097152
Pid 280096, Slab Id 4445962240
Virtual address ffff880109000000, Physical Address 109000000, Size 2097152
Pid 280096, Slab Id 4448059392
Virtual address ffff880109200000, Physical Address 109200000, Size 2097152
Pid 280215, Slab Id 471859200
Virtual address ffff88001c200000, Physical Address 1c200000, Size 2097152
Pid 280215, Slab Id 4414504960
Virtual address ffff880107200000, Physical Address 107200000, Size 2097152
Pid 280215, Slab Id 4330618880
Virtual address ffff880102200000, Physical Address 102200000, Size 2097152
Pid 280215, Slab Id 4913627136
Virtual address ffff880124e00000, Physical Address 124e00000, Size 2097152
Pid 280216, Slab Id 4383047680
Virtual address ffff880105400000, Physical Address 105400000, Size 2097152
Pid 280216, Slab Id 4328521728
Virtual address ffff880102000000, Physical Address 102000000, Size 2097152
Pid 280216, Slab Id 50331648
Virtual address ffff880003000000, Physical Address 3000000, Size 2097152
Pid 280216, Slab Id 4775215104
Virtual address ffff88011ca00000, Physical Address 11ca00000, Size 2097152
Pid 280217, Slab Id 4366270464
Virtual address ffff880104400000, Physical Address 104400000, Size 2097152
Pid 280217, Slab Id 576716800
Virtual address ffff880022600000, Physical Address 22600000, Size 2097152
Pid 280217, Slab Id 4773117952
Virtual address ffff88011c800000, Physical Address 11c800000, Size 2097152
Pid 280217, Slab Id 4911529984
Virtual address ffff880124c00000, Physical Address 124c00000, Size 2097152
Pid 280218, Slab Id 396361728
Virtual address ffff880017a00000, Physical Address 17a00000, Size 2097152
Pid 280218, Slab Id 5033164800
Virtual address ffff88012c000000, Physical Address 12c000000, Size 2097152
Pid 280218, Slab Id 377487360
Virtual address ffff880016800000, Physical Address 16800000, Size 2097152
Pid 280218, Slab Id 4347396096
Virtual address ffff880103200000, Physical Address 103200000, Size 2097152
Pid 280221, Slab Id 94371840
Virtual address ffff880005a00000, Physical Address 5a00000, Size 2097152
Pid 280221, Slab Id 4345298944
Virtual address ffff880103000000, Physical Address 103000000, Size 2097152
Pid 280221, Slab Id 4655677440
Virtual address ffff880115800000, Physical Address 115800000, Size 2097152
Pid 280221, Slab Id 255852544
Virtual address ffff88000f400000, Physical Address f400000, Size 2097152
Pid 280220, Slab Id 83886080
Virtual address ffff880005000000, Physical Address 5000000, Size 2097152
Pid 280220, Slab Id 4657774592
Virtual address ffff880115a00000, Physical Address 115a00000, Size 2097152
Pid 280220, Slab Id 4303355904
Virtual address ffff880100800000, Physical Address 100800000, Size 2097152
Pid 280220, Slab Id 257949696
Virtual address ffff88000f600000, Physical Address f600000, Size 2097152
Pid 280223, Slab Id 146800640
Virtual address ffff880008c00000, Physical Address 8c00000, Size 2097152
Pid 280223, Slab Id 4758437888
Virtual address ffff88011ba00000, Physical Address 11ba00000, Size 2097152
Pid 280223, Slab Id 4339007488
Virtual address ffff880102a00000, Physical Address 102a00000, Size 2097152
Pid 280223, Slab Id 4611637248
Virtual address ffff880112e00000, Physical Address 112e00000, Size 2097152
Pid 280226, Slab Id 4756340736
Virtual address ffff88011b800000, Physical Address 11b800000, Size 2097152
Pid 280226, Slab Id 4336910336
Virtual address ffff880102800000, Physical Address 102800000, Size 2097152
Pid 280226, Slab Id 226492416
Virtual address ffff88000d800000, Physical Address d800000, Size 2097152
Pid 280226, Slab Id 5070913536
Virtual address ffff88012e400000, Physical Address 12e400000, Size 2097152
Pid 280227, Slab Id 88080384
Virtual address ffff880005400000, Physical Address 5400000, Size 2097152
Pid 280227, Slab Id 4609540096
Virtual address ffff880112c00000, Physical Address 112c00000, Size 2097152
Pid 280227, Slab Id 4364173312
Virtual address ffff880104200000, Physical Address 104200000, Size 2097152
Pid 280224, Slab Id 4452253696
Virtual address ffff880109600000, Physical Address 109600000, Size 2097152
Pid 280224, Slab Id 148897792
Virtual address ffff880008e00000, Physical Address 8e00000, Size 2097152
Pid 280224, Slab Id 90177536
Virtual address ffff880005600000, Physical Address 5600000, Size 2097152
Pid 280224, Slab Id 461373440
Virtual address ffff88001b800000, Physical Address 1b800000, Size 2097152
Pid 280229, Slab Id 228589568
Virtual address ffff88000da00000, Physical Address da00000, Size 2097152
Pid 280229, Slab Id 5073010688
Virtual address ffff88012e600000, Physical Address 12e600000, Size 2097152
Pid 280229, Slab Id 364904448
Virtual address ffff880015c00000, Physical Address 15c00000, Size 2097152
Pid 280229, Slab Id 4424990720
Virtual address ffff880107c00000, Physical Address 107c00000, Size 2097152
Pid 280228, Slab Id 4731174912
Virtual address ffff88011a000000, Physical Address 11a000000, Size 2097152
Pid 280228, Slab Id 367001600
Virtual address ffff880015e00000, Physical Address 15e00000, Size 2097152
Pid 280228, Slab Id 4357881856
Virtual address ffff880103c00000, Physical Address 103c00000, Size 2097152
Pid 280228, Slab Id 4691329024
Virtual address ffff880117a00000, Physical Address 117a00000, Size 2097152
Pid 280225, Slab Id 463470592
Virtual address ffff88001ba00000, Physical Address 1ba00000, Size 2097152
Pid 280225, Slab Id 4733272064
Virtual address ffff88011a200000, Physical Address 11a200000, Size 2097152
Pid 280225, Slab Id 4359979008
Virtual address ffff880103e00000, Physical Address 103e00000, Size 2097152
Pid 280225, Slab Id 4590665728
Virtual address ffff880111a00000, Physical Address 111a00000, Size 2097152
Pid 280230, Slab Id 4427087872
Virtual address ffff880107e00000, Physical Address 107e00000, Size 2097152
Pid 280230, Slab Id 4397727744
Virtual address ffff880106200000, Physical Address 106200000, Size 2097152
Pid 280230, Slab Id 251658240
Virtual address ffff88000f000000, Physical Address f000000, Size 2097152
Pid 280230, Slab Id 580911104
Virtual address ffff880022a00000, Physical Address 22a00000, Size 2097152
Pid 280222, Slab Id 4487905280
Virtual address ffff88010b800000, Physical Address 10b800000, Size 2097152
Pid 280222, Slab Id 253755392
Virtual address ffff88000f200000, Physical Address f200000, Size 2097152
Pid 280222, Slab Id 578813952
Virtual address ffff880022800000, Physical Address 22800000, Size 2097152
Pid 280222, Slab Id 4949278720
Virtual address ffff880127000000, Physical Address 127000000, Size 2097152

Other services restarting

HAProxy restart finishing

[error] SalCtrl_AdfServicesStartedCheck() - : Sal Ctrl failed to start in given time

[error] do_userStart() - : Failed to start services

ADF_UIO_PROXY err: icp_adf_subsystemUnregister: Failed to shutdown subservice SAL.
[NOTICE] 106/140718 (283224) : New worker #1 (283859) forked
[NOTICE] 106/140718 (283224) : New worker #2 (283860) forked
[NOTICE] 106/140718 (283224) : New worker #3 (283861) forked
[NOTICE] 106/140718 (283224) : New worker #4 (283862) forked
[NOTICE] 106/140718 (283224) : New worker #5 (283863) forked
[NOTICE] 106/140718 (283224) : New worker #6 (283864) forked
[NOTICE] 106/140718 (283224) : New worker #7 (283865) forked
[NOTICE] 106/140718 (283224) : New worker #8 (283866) forked
[NOTICE] 106/140718 (283224) : New worker #9 (283867) forked
[NOTICE] 106/140718 (283224) : New worker #10 (283868) forked
[NOTICE] 106/140718 (283224) : New worker #11 (283869) forked
[NOTICE] 106/140718 (283224) : New worker #12 (283870) forked
[NOTICE] 106/140718 (283224) : New worker #13 (283871) forked
[NOTICE] 106/140718 (283224) : New worker #14 (283872) forked
[NOTICE] 106/140718 (283224) : New worker #15 (283873) forked
[NOTICE] 106/140718 (283224) : New worker #16 (283874) forked

cat /proc/meminfo
MemTotal:        8065500 kB
MemFree:         1070112 kB
MemAvailable:    1896840 kB
Buffers:              16 kB
Cached:          1134828 kB
SwapCached:        61704 kB
Active:           934468 kB
Inactive:        1174760 kB
Active(anon):     444884 kB
Inactive(anon):   692204 kB
Active(file):     489584 kB
Inactive(file):   482556 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554428 kB
SwapFree:       33279900 kB
Dirty:               404 kB
Writeback:             0 kB
AnonPages:        914788 kB
Mapped:            58496 kB
Shmem:            162412 kB
Slab:             189396 kB
SReclaimable:     146948 kB
SUnreclaim:        42448 kB
KernelStack:        7632 kB
PageTables:        22576 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    35490024 kB
Committed_AS:    6895048 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      395168 kB
VmallocChunk:   34358947836 kB
HardwareCorrupted:     0 kB
AnonHugePages:    520192 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      120920 kB
DirectMap2M:     8159232 kB

free -h
              total        used        free      shared  buff/cache   available
Mem:           7.7G        5.4G        1.0G        158M        1.3G        1.8G
Swap:           31G        268M         31G

cat /sys/kernel/debug/qae_mem_dbg/qae_mem_slabs
Pid 276405, Slab Id 4297064448
Virtual address ffff880100200000, Physical Address 100200000, Size 2097152
Pid 276405, Slab Id 4318035968
Virtual address ffff880101600000, Physical Address 101600000, Size 2097152
Pid 276405, Slab Id 4326424576
Virtual address ffff880101e00000, Physical Address 101e00000, Size 2097152
Pid 276405, Slab Id 171966464
Virtual address ffff88000a400000, Physical Address a400000, Size 2097152
Pid 276538, Slab Id 4596957184
Virtual address ffff880112000000, Physical Address 112000000, Size 2097152
Pid 276538, Slab Id 4607442944
Virtual address ffff880112a00000, Physical Address 112a00000, Size 2097152
Pid 276538, Slab Id 4592762880
Virtual address ffff880111c00000, Physical Address 111c00000, Size 2097152
Pid 276538, Slab Id 4649385984
Virtual address ffff880115200000, Physical Address 115200000, Size 2097152
Pid 283859, Slab Id 4443865088
Virtual address ffff880108e00000, Physical Address 108e00000, Size 2097152
Pid 283859, Slab Id 4949278720
Virtual address ffff880127000000, Physical Address 127000000, Size 2097152
Pid 283859, Slab Id 4590665728
Virtual address ffff880111a00000, Physical Address 111a00000, Size 2097152
Pid 283859, Slab Id 463470592
Virtual address ffff88001ba00000, Physical Address 1ba00000, Size 2097152
Pid 283860, Slab Id 5112856576
Virtual address ffff880130c00000, Physical Address 130c00000, Size 2097152
Pid 283860, Slab Id 228589568
Virtual address ffff88000da00000, Physical Address da00000, Size 2097152
Pid 283860, Slab Id 4445962240
Virtual address ffff880109000000, Physical Address 109000000, Size 2097152
Pid 283860, Slab Id 88080384
Virtual address ffff880005400000, Physical Address 5400000, Size 2097152
Pid 283861, Slab Id 50331648
Virtual address ffff880003000000, Physical Address 3000000, Size 2097152
Pid 283861, Slab Id 580911104
Virtual address ffff880022a00000, Physical Address 22a00000, Size 2097152
Pid 283861, Slab Id 4487905280
Virtual address ffff88010b800000, Physical Address 10b800000, Size 2097152
Pid 283861, Slab Id 4359979008
Virtual address ffff880103e00000, Physical Address 103e00000, Size 2097152
Pid 283862, Slab Id 578813952
Virtual address ffff880022800000, Physical Address 22800000, Size 2097152
Pid 283862, Slab Id 251658240
Virtual address ffff88000f000000, Physical Address f000000, Size 2097152
Pid 283862, Slab Id 4490002432
Virtual address ffff88010ba00000, Physical Address 10ba00000, Size 2097152
Pid 283862, Slab Id 146800640
Virtual address ffff880008c00000, Physical Address 8c00000, Size 2097152
Pid 283863, Slab Id 4397727744
Virtual address ffff880106200000, Physical Address 106200000, Size 2097152
Pid 283863, Slab Id 461373440
Virtual address ffff88001b800000, Physical Address 1b800000, Size 2097152
Pid 283863, Slab Id 4427087872
Virtual address ffff880107e00000, Physical Address 107e00000, Size 2097152
Pid 283863, Slab Id 83886080
Virtual address ffff880005000000, Physical Address 5000000, Size 2097152
Pid 283864, Slab Id 4383047680
Virtual address ffff880105400000, Physical Address 105400000, Size 2097152
Pid 283864, Slab Id 4448059392
Virtual address ffff880109200000, Physical Address 109200000, Size 2097152
Pid 283864, Slab Id 90177536
Virtual address ffff880005600000, Physical Address 5600000, Size 2097152
Pid 283864, Slab Id 4731174912
Virtual address ffff88011a000000, Physical Address 11a000000, Size 2097152
Pid 283865, Slab Id 253755392
Virtual address ffff88000f200000, Physical Address f200000, Size 2097152
Pid 283865, Slab Id 4424990720
Virtual address ffff880107c00000, Physical Address 107c00000, Size 2097152
Pid 283865, Slab Id 4357881856
Virtual address ffff880103c00000, Physical Address 103c00000, Size 2097152
Pid 283865, Slab Id 4655677440
Virtual address ffff880115800000, Physical Address 115800000, Size 2097152
Pid 283866, Slab Id 4450156544
Virtual address ffff880109400000, Physical Address 109400000, Size 2097152
Pid 283866, Slab Id 148897792
Virtual address ffff880008e00000, Physical Address 8e00000, Size 2097152
Pid 283866, Slab Id 4303355904
Virtual address ffff880100800000, Physical Address 100800000, Size 2097152
Pid 283866, Slab Id 4762632192
Virtual address ffff88011be00000, Physical Address 11be00000, Size 2097152
Pid 283869, Slab Id 4452253696
Virtual address ffff880109600000, Physical Address 109600000, Size 2097152
Pid 283869, Slab Id 255852544
Virtual address ffff88000f400000, Physical Address f400000, Size 2097152
Pid 283869, Slab Id 4657774592
Virtual address ffff880115a00000, Physical Address 115a00000, Size 2097152
Pid 283869, Slab Id 419430400
Virtual address ffff880019000000, Physical Address 19000000, Size 2097152
Pid 283868, Slab Id 257949696
Virtual address ffff88000f600000, Physical Address f600000, Size 2097152
Pid 283868, Slab Id 184549376
Virtual address ffff88000b000000, Physical Address b000000, Size 2097152
Pid 283868, Slab Id 400556032
Virtual address ffff880017e00000, Physical Address 17e00000, Size 2097152
Pid 283868, Slab Id 85983232
Virtual address ffff880005200000, Physical Address 5200000, Size 2097152
Pid 283867, Slab Id 4733272064
Virtual address ffff88011a200000, Physical Address 11a200000, Size 2097152
Pid 283867, Slab Id 186646528
Virtual address ffff88000b200000, Physical Address b200000, Size 2097152
Pid 283867, Slab Id 4760535040
Virtual address ffff88011bc00000, Physical Address 11bc00000, Size 2097152
Pid 283867, Slab Id 4865392640
Virtual address ffff880122000000, Physical Address 122000000, Size 2097152
Pid 283872, Slab Id 4305453056
Virtual address ffff880100a00000, Physical Address 100a00000, Size 2097152
Pid 283872, Slab Id 398458880
Virtual address ffff880017c00000, Physical Address 17c00000, Size 2097152
Pid 283872, Slab Id 4389339136
Virtual address ffff880105a00000, Physical Address 105a00000, Size 2097152
Pid 283872, Slab Id 4867489792
Virtual address ffff880122200000, Physical Address 122200000, Size 2097152
Pid 283871, Slab Id 4387241984
Virtual address ffff880105800000, Physical Address 105800000, Size 2097152
Pid 283871, Slab Id 421527552
Virtual address ffff880019200000, Physical Address 19200000, Size 2097152
Pid 283871, Slab Id 4557111296
Virtual address ffff88010fa00000, Physical Address 10fa00000, Size 2097152
Pid 283871, Slab Id 4810866688
Virtual address ffff88011ec00000, Physical Address 11ec00000, Size 2097152
Pid 283874, Slab Id 4555014144
Virtual address ffff88010f800000, Physical Address 10f800000, Size 2097152
Pid 283874, Slab Id 4812963840
Virtual address ffff88011ee00000, Physical Address 11ee00000, Size 2097152
Pid 283874, Slab Id 415236096
Virtual address ffff880018c00000, Physical Address 18c00000, Size 2097152
Pid 283874, Slab Id 4995416064
Virtual address ffff880129c00000, Physical Address 129c00000, Size 2097152
Pid 283870, Slab Id 350224384
Virtual address ffff880014e00000, Physical Address 14e00000, Size 2097152
Pid 283870, Slab Id 417333248
Virtual address ffff880018e00000, Physical Address 18e00000, Size 2097152
Pid 283870, Slab Id 4993318912
Virtual address ffff880129a00000, Physical Address 129a00000, Size 2097152
Pid 283870, Slab Id 373293056
Virtual address ffff880016400000, Physical Address 16400000, Size 2097152
Pid 283873, Slab Id 348127232
Virtual address ffff880014c00000, Physical Address 14c00000, Size 2097152
Pid 283873, Slab Id 4991221760
Virtual address ffff880129800000, Physical Address 129800000, Size 2097152
Pid 283873, Slab Id 4997513216
Virtual address ffff880129e00000, Physical Address 129e00000, Size 2097152
Pid 283873, Slab Id 4970250240
Virtual address ffff880128400000, Physical Address 128400000, Size 2097152