"/etc/init.d/qat_service status" does not list all the VFs after installing kubernetes intel qat_plugin

[irshad97feb]

I am using QAT configured system Initial status:

$ sudo /etc/init.d/qat_service status Checking status of all devices. There is 34 QAT acceleration device(s) in the system: qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:f3:00.0, #accel: 1 #engines: 9 state: up qat_dev1 - type: 4xxx, inst_id: 1, node_id: 0, bsf: 0000:f7:00.0, #accel: 1 #engines: 9 state: up qat_dev2 - type: 4xxxvf, inst_id: 0, node_id: 0, bsf: 0000:f3:00.1, #accel: 1 #engines: 1 state: up qat_dev3 - type: 4xxxvf, inst_id: 1, node_id: 0, bsf: 0000:f3:00.2, #accel: 1 #engines: 1 state: up qat_dev4 - type: 4xxxvf, inst_id: 2, node_id: 0, bsf: 0000:f3:00.3, #accel: 1 #engines: 1 state: up qat_dev5 - type: 4xxxvf, inst_id: 3, node_id: 0, bsf: 0000:f3:00.4, #accel: 1 #engines: 1 state: up qat_dev6 - type: 4xxxvf, inst_id: 4, node_id: 0, bsf: 0000:f3:00.5, #accel: 1 #engines: 1 state: up qat_dev7 - type: 4xxxvf, inst_id: 5, node_id: 0, bsf: 0000:f3:00.6, #accel: 1 #engines: 1 state: up qat_dev8 - type: 4xxxvf, inst_id: 6, node_id: 0, bsf: 0000:f3:00.7, #accel: 1 #engines: 1 state: up qat_dev9 - type: 4xxxvf, inst_id: 7, node_id: 0, bsf: 0000:f3:01.0, #accel: 1 #engines: 1 state: up qat_dev10 - type: 4xxxvf, inst_id: 8, node_id: 0, bsf: 0000:f3:01.1, #accel: 1 #engines: 1 state: up qat_dev11 - type: 4xxxvf, inst_id: 9, node_id: 0, bsf: 0000:f3:01.2, #accel: 1 #engines: 1 state: up qat_dev12 - type: 4xxxvf, inst_id: 10, node_id: 0, bsf: 0000:f3:01.3, #accel: 1 #engines: 1 state: up qat_dev13 - type: 4xxxvf, inst_id: 11, node_id: 0, bsf: 0000:f3:01.4, #accel: 1 #engines: 1 state: up qat_dev14 - type: 4xxxvf, inst_id: 12, node_id: 0, bsf: 0000:f3:01.5, #accel: 1 #engines: 1 state: up qat_dev15 - type: 4xxxvf, inst_id: 13, node_id: 0, bsf: 0000:f3:01.6, #accel: 1 #engines: 1 state: up qat_dev16 - type: 4xxxvf, inst_id: 14, node_id: 0, bsf: 0000:f3:01.7, #accel: 1 #engines: 1 state: up qat_dev17 - type: 4xxxvf, inst_id: 15, node_id: 0, bsf: 0000:f3:02.0, #accel: 1 #engines: 1 state: up qat_dev18 - type: 4xxxvf, inst_id: 16, node_id: 0, bsf: 0000:f7:00.1, #accel: 1 #engines: 1 state: up qat_dev19 - type: 4xxxvf, inst_id: 17, node_id: 0, bsf: 0000:f7:00.2, #accel: 1 #engines: 1 state: up qat_dev20 - type: 4xxxvf, inst_id: 18, node_id: 0, bsf: 0000:f7:00.3, #accel: 1 #engines: 1 state: up qat_dev21 - type: 4xxxvf, inst_id: 19, node_id: 0, bsf: 0000:f7:00.4, #accel: 1 #engines: 1 state: up qat_dev22 - type: 4xxxvf, inst_id: 20, node_id: 0, bsf: 0000:f7:00.5, #accel: 1 #engines: 1 state: up qat_dev23 - type: 4xxxvf, inst_id: 21, node_id: 0, bsf: 0000:f7:00.6, #accel: 1 #engines: 1 state: up qat_dev24 - type: 4xxxvf, inst_id: 22, node_id: 0, bsf: 0000:f7:00.7, #accel: 1 #engines: 1 state: up qat_dev25 - type: 4xxxvf, inst_id: 23, node_id: 0, bsf: 0000:f7:01.0, #accel: 1 #engines: 1 state: up qat_dev26 - type: 4xxxvf, inst_id: 24, node_id: 0, bsf: 0000:f7:01.1, #accel: 1 #engines: 1 state: up qat_dev27 - type: 4xxxvf, inst_id: 25, node_id: 0, bsf: 0000:f7:01.2, #accel: 1 #engines: 1 state: up qat_dev28 - type: 4xxxvf, inst_id: 26, node_id: 0, bsf: 0000:f7:01.3, #accel: 1 #engines: 1 state: up qat_dev29 - type: 4xxxvf, inst_id: 27, node_id: 0, bsf: 0000:f7:01.4, #accel: 1 #engines: 1 state: up qat_dev30 - type: 4xxxvf, inst_id: 28, node_id: 0, bsf: 0000:f7:01.5, #accel: 1 #engines: 1 state: up qat_dev31 - type: 4xxxvf, inst_id: 29, node_id: 0, bsf: 0000:f7:01.6, #accel: 1 #engines: 1 state: up qat_dev32 - type: 4xxxvf, inst_id: 30, node_id: 0, bsf: 0000:f7:01.7, #accel: 1 #engines: 1 state: up qat_dev33 - type: 4xxxvf, inst_id: 31, node_id: 0, bsf: 0000:f7:02.0, #accel: 1 #engines: 1 state: up

**Configuration files: sysadmin@controller-0:~$ ls -al /etc/4xx*** -rw-r----- 1 root root 5315 Jul 3 09:39 /etc/4xxx_dev0.conf -rw-r----- 1 root root 5315 Jul 3 09:39 /etc/4xxx_dev1.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev0.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev1.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev10.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev11.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev12.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev13.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev14.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev15.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev16.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev17.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev18.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev19.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev2.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev20.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev21.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev22.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev23.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev24.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev25.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev26.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev27.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev28.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev29.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev3.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev30.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev31.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev4.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev5.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev6.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev7.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev8.conf -rw-r----- 1 root root 4383 Jul 3 09:39 /etc/4xxxvf_dev9.conf

Now, If I install qat_plugin with the help of operator then as soon as qat_plugin comes up vf endpoint are not being listed when we check the qat_service status Note: Here we have not used initcontainer to provision the qat devices as we are doing after our os installation. Helm chart link

This is the status after installing intel qat_plugin and it shows only PFs not VFs. $ sudo /etc/init.d/qat_service status Checking status of all devices. There is 2 QAT acceleration device(s) in the system: qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:f3:00.0, #accel: 1 #engines: 9 state: up qat_dev1 - type: 4xxx, inst_id: 1, node_id: 0, bsf: 0000:f7:00.0, #accel: 1 #engines: 9 state: up

If I restart the qat_service menually then it gives below result:

$ sudo /etc/init.d/qat_service restart Restarting all devices. Processing /etc/4xxx_dev0.conf Processing /etc/4xxx_dev1.conf restart sriov

$ sudo /etc/init.d/qat_service status Checking status of all devices. There is 2 QAT acceleration device(s) in the system: qat_dev0 - type: 4xxx, inst_id: 0, node_id: 0, bsf: 0000:f3:00.0, #accel: 1 #engines: 9 state: up qat_dev1 - type: 4xxx, inst_id: 1, node_id: 0, bsf: 0000:f7:00.0, #accel: 1 #engines: 9 state: up

Node description:

Capacity: cpu: 64 ephemeral-storage: 10218772Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 129160204Ki pods: 110 qat.intel.com/asym-dc: 32 Allocatable: cpu: 62 ephemeral-storage: 9417620260 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 118817804Ki pods: 110 qat.intel.com/asym-dc: 32

$ lspci | egrep "4942|4943" f3:00.0 Co-processor: Intel Corporation Device 4942 (rev 40) f3:00.1 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.2 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.3 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.4 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.5 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.6 Co-processor: Intel Corporation Device 4943 (rev 40) f3:00.7 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.0 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.1 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.2 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.3 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.4 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.5 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.6 Co-processor: Intel Corporation Device 4943 (rev 40) f3:01.7 Co-processor: Intel Corporation Device 4943 (rev 40) f3:02.0 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.0 Co-processor: Intel Corporation Device 4942 (rev 40) f7:00.1 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.2 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.3 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.4 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.5 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.6 Co-processor: Intel Corporation Device 4943 (rev 40) f7:00.7 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.0 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.1 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.2 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.3 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.4 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.5 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.6 Co-processor: Intel Corporation Device 4943 (rev 40) f7:01.7 Co-processor: Intel Corporation Device 4943 (rev 40) f7:02.0 Co-processor: Intel Corporation Device 4943 (rev 40)

Please let us know whether this behaviour is expected or not. Thanks

[mythi]

See #79

[jdschuet]

Please refer to: https://intel.github.io/quickassist/AppNotes/Containers/index.html#intel-qat-in-linux-containers for additional details on running QAT within a container.

[fionatrahe]

Closing this as the discussion is covered in #79