Open svenschwermer opened 5 months ago
Apologies for the late reply!
Right, if I remember correctly the library is built against 1.1.0 (see dependency).
OpenSSL 3 introduces a bunch of changes (including the transition from "engine" to "provider"), so the build errors you have seen seem reasonable, unfortunately. Keeping this open as a feature request for future consideration. We should probably also mention this somewhere in our docs, as you noted.
I'm getting similar core dumps, making it completely unusable on latest Fedora 40.
I tried to compile the latest master branch against openssl3 using :
bazel build --config openssl //kmsp11/main:libkmsp11.so
but it complained of conflicts between BoringSSL-openssl1 compat headers and Openssl3.
Then I tried to recompile the latest master branch without modifications and the build succeeded.
However when I try to sign something, I get this stacktrace:
#0 0x00007fe7c81ca0d2 in pkcs11_getattr_alloc (ctx=ctx@entry=0x3f591e987bb78c92, session=4802441702199765720,
object=object@entry=7857815905065540909, type=type@entry=288, value=value@entry=0x7ffc7285e5e0,
size=size@entry=0x7ffc7285e5d8) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:62
#1 0x00007fe7c81ca8b0 in pkcs11_getattr_bn (ctx=ctx@entry=0x3f591e987bb78c92, session=<optimized out>,
object=object@entry=7857815905065540909, type=type@entry=288, bn=bn@entry=0x7ffc7285e640)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:92
#2 0x00007fe7c81d07df in pkcs11_get_rsa (key=0x70bda0)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:197
#3 0x00007fe7c81d0b50 in pkcs11_get_evp_key_rsa (key=0x70bda0)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:265
#4 0x00007fe7c81cea12 in pkcs11_get_key (key0=key0@entry=0x70bda0, object_class=<optimized out>)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_key.c:456
#5 0x00007fe7c81ceaaa in pkcs11_rsa (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:34
#6 pkcs11_get_key_size (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:332
#7 pkcs11_private_encrypt (padding=1, key=0x70bda0, to=0x72ae50 "\232\326o",
from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005", flen=51)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:91
#8 pkcs11_rsa_priv_enc_method (flen=51, from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005",
to=0x72ae50 "\232\326o", rsa=<optimized out>, padding=1)
at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:384
#9 0x00007fe7c7dbfd86 in RSA_sign (type=<optimized out>,
m=m@entry=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n",
m_len=m_len@entry=32, sigret=sigret@entry=0x72ae50 "\232\326o", siglen=siglen@entry=0x7ffc7285eb44,
rsa=rsa@entry=0x6f7820) at crypto/rsa/rsa_sign.c:307
#10 0x00007fe7c7dc2a91 in pkey_rsa_sign (ctx=0x6ff9c0, sig=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50,
tbs=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n", tbslen=32)
at crypto/rsa/rsa_pmeth.c:178
#11 0x00007fe7c7d4f91b in EVP_DigestSignFinal (ctx=<optimized out>, sigret=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50)
at crypto/evp/m_sigver.c:677
#12 0x00007fe7c7da0e04 in PKCS7_SIGNER_INFO_sign (si=si@entry=0x70d780) at crypto/pkcs7/pk7_doit.c:934
#13 0x00007fe7c7da2025 in do_pkcs7_signed_attrib (mctx=<optimized out>, si=0x70d780) at crypto/pkcs7/pk7_doit.c:711
#14 PKCS7_dataFinal (p7=p7@entry=0x6f02f0, bio=bio@entry=0x5e87c0) at crypto/pkcs7/pk7_doit.c:833
#15 0x0000000000403103 in IDC_set (p7=p7@entry=0x6f02f0, si=si@entry=0x70d780, image=<optimized out>) at idc.c:216
#16 0x0000000000402947 in main (argc=<optimized out>, argv=<optimized out>) at sbsign.c:274
I works fine on Fedora 39:
openssl-pkcs11-0.4.12-4.fc39.x86_64
openssl-libs-3.1.1-4.fc39.x86_64
But fails as described above with Fedora 40:
openssl-libs-3.2.1-2.fc40.x86_64
openssl-pkcs11-0.4.12-8.fc40.x86_64
Apparently the engines in OpenSSL3 are still supposed to work, and the migration to providers instead is not necessary right this minute.
Note: the version of openssl on the machine should not change anything, as this is a pkcs11 library that could be called by anything (not necessarily OpenSSL). In my case kmspkcs11 is called by p11kit which is called by openssl engine.
Using libkmsp11.so (version 1.3) with OpenSSL 3 doesn't appear to work. I'm getting aborts and core dumps (see attached)
I also couldn't build the latest master against OpenSSL 3. I couldn't find any documentation that OpenSSL 1.x is required :shrug: