a-dma
commented
3 years ago I believe you are not correctly matching the hashes between encryption and decryption. Keep in mind that there are two hash functions at play, one for the MGF1 function and one for OAEP.
Here are some examples:
First some data
$ cat data.txt
hello worldUsing SHA256 for both:
$ openssl pkeyutl -in data.txt -encrypt -pubin -inkey pub.pem -keyform PEM -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256 -out data.txt.enc256
$ pkcs11-tool --module ./libykcs11.so --decrypt -m RSA-PKCS-OAEP --id 1 -i /tmp/data.txt.enc256 --mgf MGF1-SHA256 --hash-algorithm SHA256
Using slot 0 with a present token (0x0)
Logging in to "YubiKey PIV #123456".
Please enter User PIN:
Using decrypt algorithm RSA-PKCS-OAEP
OAEP parameters: hashAlg=SHA256, mgf=MGF1-SHA256, source_type=0, source_ptr=(nil), source_len=0
hello worldUsing SHA384 for both:
$ openssl pkeyutl -in data.txt -encrypt -pubin -inkey pub.pem -keyform PEM -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha384 -pkeyopt rsa_mgf1_md:sha384 -out data.txt.enc384
$ pkcs11-tool --module ./libykcs11.so --decrypt -m RSA-PKCS-OAEP --id 1 -i /tmp/data.txt.enc384 --mgf MGF1-SHA384 --hash-algorithm SHA384
Using slot 0 with a present token (0x0)
Logging in to "YubiKey PIV #123456".
Please enter User PIN:
Using decrypt algorithm RSA-PKCS-OAEP
OAEP parameters: hashAlg=SHA384, mgf=MGF1-SHA384, source_type=0, source_ptr=(nil), source_len=0
hello worldUsing SHA256 for the OAEP hash and SHA384 for the MGF1 hash:
$ openssl pkeyutl -in data.txt -encrypt -pubin -inkey pub.pem -keyform PEM -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha384 -out data.txt.enc256-mgf384
$ pkcs11-tool --module ./libykcs11.so --decrypt -m RSA-PKCS-OAEP --id 1 -i /tmp/data.txt.enc256-mgf384 --mgf MGF1-SHA384 --hash-algorithm SHA256
Using slot 0 with a present token (0x0)
Logging in to "YubiKey PIV #123456".
Please enter User PIN:
Using decrypt algorithm RSA-PKCS-OAEP
OAEP parameters: hashAlg=SHA256, mgf=MGF1-SHA384, source_type=0, source_ptr=(nil), source_len=0
hello worldAnd finally your last example, in the "update". You're using openssl rsautl which defaults to using SHA1 for both hashes but you are using SHA256 for both hashes during decryption. Correctly setting the algorithms produces the expected result:
$ openssl rsautl -encrypt -oaep -keyform PEM -pubin -inkey pub.pem -in data.txt -out data.txt.enc
$ pkcs11-tool --module ./ykcs11/libykcs11.so --decrypt -m RSA-PKCS-OAEP --id 1 -i /tmp/data.txt.enc --mgf MGF1-SHA1 --hash-a SHA-1
Using slot 0 with a present token (0x0)
Logging in to "YubiKey PIV #123456".
Please enter User PIN:
Using decrypt algorithm RSA-PKCS-OAEP
OAEP parameters: hashAlg=SHA-1, mgf=MGF1-SHA1, source_type=0, source_ptr=(nil), source_len=0
hello world
mouse07410
As you can see, RSA-OAEP decryption by YKCS11 library fails. OpenSSL using OpenSC and libp11 engine, decrypts that same file OK (and correctly, as the decrypted data matches the original plaintext).
SPY capture:
Update
Here's your own test failing: