Where is the KERNEL_DIR in driver/makefile please?

[Yogurt2019]

As a beginner it's really hard to find out how to use this And finally i know how to build this under petalinux, but don't know how to launch hw acceleration. I found that the modprobe cryptodev not exist, and maybe the .ko file is generated by driver/makefile, but I don't know where the KERNEL_DIR variable is. Any help would be appreciated !

[ovpanait]

Hi @Yogurt2019,

Thanks for your interest in this project!

The KERNEL_DIR environment variable needs to be set to the path of the kernel sources in your Petalinux build.

I personally do not use Petalinux to test this project because I am not familiar with it, so I usually compile the kernel and driver manually, something like:

Get sources

git clone https://github.com/ovpanait/zynq-aes.git git clone --depth 1 --branch xlnx_rebase_v5.4 https://github.com/Xilinx/linux-xlnx.git

Get toolchain

wget https://releases.linaro.org/components/toolchain/binaries/latest-7/arm-linux-gnueabihf/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz tar -xvf gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz

Set up environment

mkdir linux-modules export CROSS_COMPILE="$(pwd)/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-" export KERNEL_DIR="$(pwd)/linux-xlnx" export ARCH=arm

Build kernel

cd ${KERNEL_DIR} make xilinx_zynq_defconfig cat >> .config <<EOF CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_CTR=y CONFIG_CRYPTO_CFB=y CONFIG_CRYPTO_OFB=y CONFIG_CRYPTO_PCBC=y CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_ENGINE=m CONFIG_CRYPTO_USER_API=y CONFIG_CRYPTO_USER_API_SKCIPHER=y EOF

make olddefconfig make uImage -j4 LOADADDR="0x10000000" make modules -j4 make modules_install INSTALL_MOD_PATH=../linux-modules cd -

Build zynq-aes driver

cd zynq-aes/driver make clean make -j4 make -C tests cd -

Optionally, build cryptodev

git clone https://github.com/cryptodev-linux/cryptodev-linux.git cd cryptodev-linux make -j4 cd -

The last thing you need to do is to recompile your dtb to include the dts fragments from https://github.com/ovpanait/zynq-aes/blob/master/driver/pl.dtsi

Place the kernel, modules, bitstream and dtb on your board and run a tcrypt test.

If you can put together a list of steps to integrate everything with petalinux, I would be more than happy to include it in README!

Thanks, Ovidiu

[Yogurt2019]

Thanks for this detailed answer!It's really appreciated. My board is ZYNQ AX7020. After days of learning, I thinks it's almost done with petalinux, but a little error. When I run modprobe -l, I can find the zynqaes_crypto.ko, cryptodev.ko and crypto_engine.ko, after modprobe zynqaes_crypto and other .ko files, OpenSSL gives this error:

root@ax_peta:~/sd/openssl-1.1.1g# openssl speed -evp aes-128-cbc -engine devcrypto invalid engine "devcrypto" 3069685776:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:../openssl-1.1.1b/crypto/dso/dso_dlfcn.c:119:filename(/usr/lib/engines-1.1/devcrypto.so): /usr/lib/engines-1.1/devcrypto.so: cannot open shared object file: No such file or directory 3069685776:error:25070067:DSO support routines:DSO_load:could not load the shared library:../openssl-1.1.1b/crypto/dso/dso_lib.c:162: 3069685776:error:260B6084:engine routines:dynamic_load:dso not found:../openssl-1.1.1b/crypto/engine/eng_dyn.c:414: 3069685776:error:2606A074:engine routines:ENGINE_by_id:no such engine:../openssl-1.1.1b/crypto/engine/eng_list.c:334:id=devcrypto 3069685776:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:../openssl-1.1.1b/crypto/dso/dso_dlfcn.c:119:filename(libdevcrypto.so): libdevcrypto.so: cannot open shared object file: No such file or directory 3069685776:error:25070067:DSO support routines:DSO_load:could not load the shared library:../openssl-1.1.1b/crypto/dso/dso_lib.c:162: 3069685776:error:260B6084:engine routines:dynamic_load:dso not found:../openssl-1.1.1b/crypto/engine/eng_dyn.c:414:

And the speed is just slow, without hw acceleration. Which version of OpenSSL are you using? When I get the zynqaes_crypto.ko file, how can I use it on board to exec hw acceleration?

Thank you very much!

[ovpanait]

Hi @Yogurt2019,

First, check if the zynqaes_crypto probe function was called successfully after loading the module:

root@arty-zynq7:~# modprobe crypto-engine
root@arty-zynq7:~# insmod zynqaes_crypto.ko
zynqaes axidmatest@1: will run requests pump with realtime priority
root@arty-zynq7:~# cat /proc/crypto | grep -i zynqaes
driver       : zynqaes-gcm
module       : zynqaes_crypto
driver       : zynqaes-ofb
module       : zynqaes_crypto
driver       : zynqaes-cfb
module       : zynqaes_crypto
driver       : zynqaes-ctr
module       : zynqaes_crypto
driver       : zynqaes-pcbc
module       : zynqaes_crypto
driver       : zynqaes-cbc
module       : zynqaes_crypto
driver       : zynqaes-ecb
module       : zynqaes_crypto

If /proc/crypto does not contain any zynqaes-* implementations, it indicates a problem with the dtb.

Also, you need an openssl version configured with "enable-devcryptoeng" flag. Afterwards, you just load the cryptodev module and openssl will automatically use the correct engine:

root@arty-zynq7:~#  insmod cryptodev.ko
root@arty-zynq7:~#  openssl speed -evp aes-128-cbc -elapsed

[Yogurt2019]

Sorry to bother you with so many questions, but I'm still not able to run cryptodev on my board. It's said on cryptodev github page that a patch is needed by cryptodev on openssl, but the patch page cannot be opened. I've compiled OpenSSL for many times and the question is still not solved. So I wonder what version of OpenSSL are you using? What version of Cryptodev are you using? Maybe a more detailed instruction is needed for me. Thanks a lot!

[ovpanait]

I use Yocto to compile everything for my board (as described in the last README section), so the steps might not be applicable for your board.

I would suggest testing with tcrypt kernel module, rather than openssl:

1. compile the kernel with

   CONFIG_CRYPTO_TEST=m

2. root@arty-zynq7:~# modprobe tcrypt mode=500 sec=1
   tcrypt:
   testing speed of async ecb(aes) (zynqaes-ecb) encryption
   tcrypt: test 0 (128 bit key, 16 byte blocks): 15824 operations in 1 seconds (253184 bytes)
   tcrypt: test 1 (128 bit key, 64 byte blocks): 15958 operations in 1 seconds (1021312 bytes)
   tcrypt: test 2 (128 bit key, 256 byte blocks): 15721 operations in 1 seconds (4024576 bytes)
   tcrypt: test 3 (128 bit key, 1024 byte blocks): 14125 operations in 1 seconds (14464000 bytes)
   tcrypt: test 4 (128 bit key, 1472 byte blocks): 14057 operations in 1 seconds (20691904 bytes)
   ...

I'll try to put together some steps to compile openssl with cryptodev support manually, maybe it will help you.

[Yogurt2019]

I can finnally run tcrypt on my board, thx a lot. Integration with Petalinux is completely OK for this project, and is simpler than manually build (for me).

Integration with Petalinux:

1.Make this project, generate .xsa file with vivado Files -> Export -> Export Hardware and include bitstream.If project can't make, comment the first line of tools/create_ip.tcl(source $emv(TCL_INCLUDE)). Maybe you have the debug.tcl, but it's not in this project.

2.Create Petalinux project with

petalinux-create --type project --template zynq --name zynq_aes

3.Config Petalinux hardware info

petalinux-config --get-hw-description ../xsa

../xsa stores your .xsa file. 4.Create custom module in Petalinux

petalinux-create -t modules --name zynqaes --enable

5.Copy files in this project/drivers to Petalinux project/project-spec/meta-user/recipes-modules/zynqaes/files, overwrite the Makefile, and most important, change the KERNEL_DIR to KERNEL_SRC in Makefile. Add the new added files into zynqaes.bb SRC_URI. It should look like:

SRC_URI = "file://Makefile \ file://zynqaes.c \ file://zynqaes.h \ file://zynqaes_aead.c \ file://zynqaes_skcipher.c \ file://COPYING \ "

6.Copy the content of drivers/pl.dtsi into Petalinux project/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi to add this project module to device tree.

7.Add the crypto configs into Petalinux project/components/plnx_workspace/sources/linux-xlnx/.config or .config.new.

CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_CTR=y CONFIG_CRYPTO_CFB=y CONFIG_CRYPTO_OFB=y CONFIG_CRYPTO_PCBC=y CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_ENGINE=m CONFIG_CRYPTO_USER_API=y CONFIG_CRYPTO_USER_API_SKCIPHER=y

8.petalinux-build

9.Copy files into SD card, run:

modprobe crypto-engine

10.Run tcrypt test:

modprobe tcrypt mode=500 sec=1

11.Enable hardware acceleration (this project)

modprobe zynqaes-crypto

12.Run tcrypt test:

modprobe tcrypt mode=500 sec=1

You should see the difference.

Really learned a lot from running this project, and thx again for your help!

[Yogurt2019]

And I got this:

testing speed of async cbc(aes) (zynqaes-cbc) decryption tcrypt: test 0 (128 bit key, 16 byte blocks): 18997 operations in 1 seconds (303952 bytes) tcrypt: test 1 (128 bit key, 64 byte blocks): 18794 operations in 1 seconds (1202816 bytes) tcrypt: test 2 (128 bit key, 256 byte blocks): 18523 operations in 1 seconds (4741888 bytes) tcrypt: test 3 (128 bit key, 1024 byte blocks): 17544 operations in 1 seconds (17965056 bytes) tcrypt: test 4 (128 bit key, 8192 byte blocks): 7703 operations in 1 seconds (63102976 bytes) tcrypt: test 5 (192 bit key, 16 byte blocks): tcrypt: setkey() failed flags=0 tcrypt: failed to load transform for lrw(aes): -2 tcrypt: failed to load transform for lrw(aes): -2 tcrypt: failed to load transform for xts(aes): -2 tcrypt: failed to load transform for xts(aes): -2 tcrypt: failed to load transform for cts(cbc(aes)): -2 tcrypt: failed to load transform for cts(cbc(aes)): -2

And when I run drivers/tests/stress I got this:

---- Running testcase for aead, gcm(aes), 16 bytes key ----

bind: No such file or directory Failed to run testcase for aead, gcm(aes), 16 bytes key

FAIL: testcase aead, gcm(aes), 16 bytes key

Is this right? Did you get this problem? I don't know if it is the ETH problem on my board. Can you give me a result on running those commands? Thx.

[ovpanait]

I can finnally run tcrypt on my board, thx a lot. Integration with Petalinux is completely OK for this project, and is simpler than manually build (for me).

Integration with Petalinux:

1.Make this project, generate .xsa file with vivado Files -> Export -> Export Hardware and include bitstream.If project can't make, comment the first line of tools/create_ip.tcl(source $emv(TCL_INCLUDE)). Maybe you have the debug.tcl, but it's not in this project.

2.Create Petalinux project with

petalinux-create --type project --template zynq --name zynq_aes

3.Config Petalinux hardware info

petalinux-config --get-hw-description ../xsa

../xsa stores your .xsa file. 4.Create custom module in Petalinux

petalinux-create -t modules --name zynqaes --enable

5.Copy files in this project/drivers to Petalinux project/project-spec/meta-user/recipes-modules/zynqaes/files, overwrite the Makefile, and most important, change the KERNEL_DIR to KERNEL_SRC in Makefile. Add the new added files into zynqaes.bb SRC_URI. It should look like:

SRC_URI = "file://Makefile \ file://zynqaes.c \ file://zynqaes.h \ file://zynqaes_aead.c \ file://zynqaes_skcipher.c \ file://COPYING \ "

6.Copy the content of drivers/pl.dtsi into Petalinux project/project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi to add this project module to device tree.

7.Add the crypto configs into Petalinux project/components/plnx_workspace/sources/linux-xlnx/.config or .config.new.

CONFIG_CRYPTO_ECB=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_CTR=y CONFIG_CRYPTO_CFB=y CONFIG_CRYPTO_OFB=y CONFIG_CRYPTO_PCBC=y CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_ENGINE=m CONFIG_CRYPTO_USER_API=y CONFIG_CRYPTO_USER_API_SKCIPHER=y

8.petalinux-build

9.Copy files into SD card, run:

modprobe crypto-engine

10.Run tcrypt test:

modprobe tcrypt mode=500 sec=1

11.Enable hardware acceleration (this project)

modprobe zynqaes-crypto

12.Run tcrypt test:

modprobe tcrypt mode=500 sec=1

You should see the difference.

Really learned a lot from running this project, and thx again for your help!

Thanks a lot for the feedback! I will test these steps and update the documentation to include these. Also, the "debug.tcl" seems to be a host pollution issue on my side, thanks for reporting it!

[ovpanait]

And I got this:

testing speed of async cbc(aes) (zynqaes-cbc) decryption tcrypt: test 0 (128 bit key, 16 byte blocks): 18997 operations in 1 seconds (303952 bytes) tcrypt: test 1 (128 bit key, 64 byte blocks): 18794 operations in 1 seconds (1202816 bytes) tcrypt: test 2 (128 bit key, 256 byte blocks): 18523 operations in 1 seconds (4741888 bytes) tcrypt: test 3 (128 bit key, 1024 byte blocks): 17544 operations in 1 seconds (17965056 bytes) tcrypt: test 4 (128 bit key, 8192 byte blocks): 7703 operations in 1 seconds (63102976 bytes) tcrypt: test 5 (192 bit key, 16 byte blocks): tcrypt: setkey() failed flags=0

Currently, the setkey() fails with 192-bit keys because the driver does not implement a fallback for this scenario (the hw engine only supports 128/256-bit keys, so for 192-bit keys we just return an error in the driver). This should be fixed if the driver implements a fallback. It's on my TODO list.

tcrypt: failed to load transform for lrw(aes): -2 tcrypt: failed to load transform for lrw(aes): -2 tcrypt: failed to load transform for xts(aes): -2 tcrypt: failed to load transform for xts(aes): -2 tcrypt: failed to load transform for cts(cbc(aes)): -2 tcrypt: failed to load transform for cts(cbc(aes)): -2

And when I run drivers/tests/stress I got this:

---- Running testcase for aead, gcm(aes), 16 bytes key ---- bind: No such file or directory Failed to run testcase for aead, gcm(aes), 16 bytes key FAIL: testcase aead, gcm(aes), 16 bytes key

Some GCM kernel config needs to be enabled, but I don't remember exactly which one. I will sort this out tonight and let you know.

Is this right? Did you get this problem? I don't know if it is the ETH problem on my board. Can you give me a result on running those commands? Thx.

[ovpanait]

---- Running testcase for aead, gcm(aes), 16 bytes key ---- bind: No such file or directory Failed to run testcase for aead, gcm(aes), 16 bytes key FAIL: testcase aead, gcm(aes), 16 bytes key

Is this right? Did you get this problem?

Could you test again with the following kernel configs enabled?

CONFIG_CRYPTO_AEAD=y CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_USER_API_AEAD=y

[Yogurt2019]

I tested these and passed the aead test. I didn't set CONFIG_CRYPTO_USER_API_AEAD before, so it didn't work. Waiting for your cryptodev tortorial, and thanks a lot.