OpenSSL3 provider support

[patzol768]

Hi Everyone, here are some changes to begin supporting OpenSSL3 providers.

Key generation, sign + verify, encrypt + decrypt are available, though requires a lot-lot-lot of additional testing and maybe some fixes. Please check the prov-openssl.sh in tests. I have no access to real HSM devices, hence everything was made with Opencrytoki's SW token.

[mtrojnar]

This PR is too large to merge as a single unit. Please split it into individual features, beginning with the libp11 API extension, and submit each feature as a separate pull request.

[mtrojnar]

@patzol768 Are you still interested in getting this feature merged? It would be great to have it merged, but there are way too many unrelated changes in this PR. I'm fine with merging an unstable new feature as long as it doesn't break the existing functionalities, which are used for business-critical applications. Therefore, any changes to the existing code need to be justified, reviewed and thoroughly tested.

[dengert]

Nice to see mods to libp11 to use OpenSSL 3 providers as engines will be deprecated some time soon. There is at least one other project that is developing a PKCS11 provider: https://github.com/latchset/pkcs11-provider It is quite active with multiple contributors.

I would suggest that any names for modules or env names or used in opensc,cnf files, include "libp11" or "LIBP11"in the names. For example The name pkcs11prov could be libp11prov or `pkcs11-libp11prov' of something to
The engines and providers already are very confusing.

I am building on Ubuntu-24.04 with OpenSSL 3.3.2 to install in /opt/ossl-3.3.2/ and note that OpenSSL installs modules in
/opt/ossl-3.3.1/lib/ossl-modules legacy.so' but this PR installspkcs11prov.soin/opt/ossl-3.3.1/lib.configure.acsome code to install engines in engines-3. Can the configure be changed to store intoossl-modules`?

PKCS11FORCELOGIN=1 \
FORCELOGIN=1 \
PKCS11MODULE=/opt/ossl-active/lib/opensc-pkcs11.so \
PKCS11VERBOSE=1 \
VERBOSE=1 \
OPENSSL_MODULES=/opt/ossl-active/lib/ossl-modules /opt/ossl-active/bin/openssl req -verbose -provider legacy -provider pkcs11prov -sha256 -new -key 'pkcs11prov:type=private;id=01;pin-value=123456' -out cards/8103.myreq.9A.pem -text
Using configuration from /tmp/genreq.131011.openssl.conf
Could not open file or uri for loading private key from pkcs11prov:type=private;id=01;pin-value=123456
801BAD21857B0000:error:80000002:system library:file_open:No such file or directory:../openssl/providers/implementations/storemgmt/file_store.c:263:calling stat(pkcs11prov:type=private;id=01;pin-value=123456)
801BAD21857B0000:error:1608010C:STORE routines:inner_loader_fetch:unsupported:../openssl/crypto/store/store_meth.c:360:No store loader found. For standard store loaders you need at least one of the default or base providers available. Did you forget to load them? Info: Global default library context, Scheme (pkcs11prov : 0), Properties (<null>)

The opensc-debug.log shows opensc-pkcs11.so was loaded, and accessed the card and last PKCS11 line was: "C_GetTokenInfo: C_GetTokenInfo(0) returns CKR_OK"

Looks like OpenSSL is expecting pkcs11prov to have a store loader

[dengert]

And a rebase of this PR would be good.

[dengert]

I have gotten further using gdb --args /opt/ossl-3.3.1/bin/openssl req -verbose -provider pkcs11prov -provider default -sha256 -new -key pkcs11://pkcs11:type=private\;id=%01 -out cards/8103.myreq.9A.pem -text

Thread 1 "openssl" hit Breakpoint 1, EVP_MD_CTX_copy_ex (out=0x5555556fcb20, in=0x5555556f4d80) at ../openssl/crypto/evp/digest.c:643
643             ERR_raise(ERR_LIB_EVP, EVP_R_NOT_ABLE_TO_COPY_CTX);
(gdb) where
#0  EVP_MD_CTX_copy_ex (out=0x5555556fcb20, in=0x5555556f4d80) at ../openssl/crypto/evp/digest.c:643
#1  0x00007ffff7a1265c in EVP_DigestSignFinal (ctx=0x5555556f4d80, sigret=0x5555556eb260 "k\360=", siglen=0x7fffffffd048) at ../openssl/crypto/evp/m_sigver.c:550
#2  0x00007ffff78d6ff1 in ASN1_item_sign_ctx (it=<optimized out>, algor1=<optimized out>, algor1@entry=0x5555556eaa98, algor2=<optimized out>, algor2@entry=0x0, signature=0x5555556fc440, data=data@entry=0x5555556eaa60, ctx=ctx@entry=0x5555556f4d80) at ../openssl/crypto/asn1/a_sign.c:275
#3  0x00007ffff7bab9de in X509_REQ_sign_ctx (x=x@entry=0x5555556eaa60, ctx=ctx@entry=0x5555556f4d80) at ../openssl/crypto/x509/x_all.c:141
#4  0x0000555555604344 in do_X509_REQ_sign (x=x@entry=0x5555556eaa60, pkey=pkey@entry=0x5555556a2c80, md=md@entry=0x7fffffffdf6c "sha256", sigopts=sigopts@entry=0x0) at ../openssl/apps/lib/apps.c:2372
#5  0x00005555555d2841 in req_main (argc=<optimized out>, argv=<optimized out>) at ../openssl/apps/req.c:886
#6  0x00005555555c70a1 in do_cmd (prog=prog@entry=0x55555567b0d0, argc=argc@entry=13, argv=argv@entry=0x7fffffffdac0) at ../openssl/apps/openssl.c:426
#7  0x000055555559ed78 in main (argc=<optimized out>, argv=<optimized out>) at ../openssl/apps/openssl.c:307
(gdb) p in
$14 = (const EVP_MD_CTX *) 0x5555556f4d80
(gdb) p *in
$15 = {reqdigest = 0x7ffff7d15500 <sha256_md>, digest = 0x5555556e24f0, engine = 0x0, flags = 0, md_data = 0x0, pctx = 0x5555556792f0, update = 0x0, algctx = 0x5555556fc8c0, fetched_digest = 0x5555556e24f0}
(gdb) p *in.digest
$16 = {type = 672, pkey_type = 0, md_size = 32, flags = 0, origin = 0, init = 0x0, update = 0x0, final = 0x0, copy = 0x0, cleanup = 0x0, block_size = 64, ctx_size = 0, md_ctrl = 0x0, name_id = 102, type_name = 0x5555556e9ff0 "SHA2-256", description = 0x0, prov = 0x55555567a150, refcnt = {val = 4}, newctx = 0x7ffff7e815e7 <p11_digest_SHA256_newctx>, dinit = 0x7ffff7e81075 <p11_digest_init>, dupdate = 0x7ffff7e81130 <p11_digest_update>, dfinal = 0x7ffff7e811bf <p11_digest_final>, dsqueeze = 0x0, digest = 0x7ffff7e8160a <p11_digest_SHA256_digest>, freectx = 0x7ffff7e80ebe <p11_digest_freectx>, dupctx = 0x7ffff7e80f70 <p11_digest_dupctx>, get_params = 0x7ffff7e81663 <p11_digest_SHA256_get_params>, set_ctx_params = 0x0, get_ctx_params = 0x0, gettable_params = 0x7ffff7e81449 <p11_digest_gettable_params>, settable_ctx_params = 0x0, gettable_ctx_params = 0x0}

This appears as the provider is calling the pkcs11 module to do the SHA256 digest for the CSR.

3985 P:137512; T:0x140737352706944 16:28:20.126 [opensc-pkcs11] pkcs15-pin.c:730:sc_pkcs15_get_pin_info: returning with: 0 (Success)
3986 P:137512; T:0x140737352706944 16:28:20.126 [opensc-pkcs11] pkcs11-session.c:303:C_GetSessionInfo: C_GetSessionInfo(0x5555556f6810) = CKR_OK
3987 P:137512; T:0x140737352706944 16:28:20.126 [opensc-pkcs11] pkcs11-object.c:543:C_DigestInit: C_DigestInit(hSession=0x5555556f6810)
3988 P:137512; T:0x140737352706944 16:28:20.126 [opensc-pkcs11] mechanism.c:299:sc_pkcs11_md_init: called
3989 P:137512; T:0x140737352706944 16:28:20.126 [opensc-pkcs11] misc.c:268:session_start_operation: called
3990 P:137512; T:0x140737352706944 16:28:20.127 [opensc-pkcs11] misc.c:269:session_start_operation: Session 0x5555556f6810, type 3
3991 P:137512; T:0x140737352706944 16:28:20.127 [opensc-pkcs11] mechanism.c:319:sc_pkcs11_md_init: returning with: 0 (Success)
3992 P:137512; T:0x140737352706944 16:28:20.127 [opensc-pkcs11] pkcs11-object.c:548::C_DigestInit: C_DigestInit() = CKR_OK
3993 P:137512; T:0x140737352706944 16:34:30.435 [opensc-pkcs11] misc.c:290:session_get_operation: called
3994 P:137512; T:0x140737352706944 16:34:30.435 [opensc-pkcs11] mechanism.c:339:sc_pkcs11_md_update: returning with: 0 (Success)
3995 P:137512; T:0x140737352706944 16:34:30.436 [opensc-pkcs11] pkcs11-object.c:615:C_DigestUpdate: C_DigestUpdate() = CKR_OK
3996 P:137512; T:0x140737352706944 16:35:25.097 [opensc-pkcs11] pkcs11-session.c:58:C_OpenSession: C_OpenSession(0x0)
3997 P:137512; T:0x140737352706944 16:35:25.097 [opensc-pkcs11] slot.c:472:slot_get_token: Slot(id=0x0): get token
3998 P:137512; T:0x140737352706944 16:35:25.097 [opensc-pkcs11] slot.c:490:slot_get_token: Slot-get-token returns OK
3999 P:137512; T:0x140737352706944 16:35:25.097 [opensc-pkcs11] pkcs11-session.c:94:C_OpenSession: C_OpenSession handle: 0x5555556fcb70
4000 P:137512; T:0x140737352706944 16:35:25.097 [opensc-pkcs11] pkcs11-session.c:97:C_OpenSession: C_OpenSession() = CKR_OK

And that is as much of the opensc-debug.log that has been written to disk. What looks strange is extra calls to C_OpenSession as C_DigestInit, C_DigestUpdate and C_DigestFinal (which is not in the log) should be run using the same session. This looks like a bug.

The OpenSC PKCS11 module lists these mechanisms, note that since there is no "hw" bit set, the pkcs11 module is doing the operation in software, i.e. CK_MECHANISM_INFO flag CKF_HW is not set. So should the digests be done by OpenSSL i.e. default-provider rather then the pkcs11-provider?

[dengert]

Using OpenSSL 3.3.1 and adding digest_disabled = 1 and rand_disabled = 1 to openssl.cnf a CSR was created using OpenSC module with a PIV card!

[mouse07410]

Using OpenSSL 3.3.1 and adding digest_disabled = 1 and rand_disabled = 1 to openssl.cnf a CSR was created using OpenSC module with a PIV card!

You mean - adding to the pkcs11 provider section? Perhaps, posting here a direct quote from your openssl.cnf would be useful?

[dengert]

@mouse07410, I just got it working once and need to clean up what I did, as the openssl.cnf file was based on 1.1.x to be used only for openssl req called from some scripts I used to generate CSR for test PIV cards: 9A, 9C, 9D and 9E which differ in usage and extensions.

I basically followed: https://github.com/patzol768/libp11/tree/master?tab=readme-ov-file#pkcs-11-module-configuration---openssl3-provider and used a combination of ENV variables and provider params found here: https://github.com/patzol768/libp11/blob/master/src/prov_front.c#L204-L218 and here: https://github.com/patzol768/libp11/blob/master/src/prov_front.c#L150-L176 As noted in the previous comments, I had to add

The build built the engine and installed in DESTDIR/lib/engines-3/pkcs11.so and libpkcs11.so -> pkcs11.so and built the provider and installed in DESTDIR/lib/pkcs11prov.so but OpenSSL installs other providers in DESTDIR/lib/openssl-modules So I copied pkcs11prov.so to DESTDIR/lib/openssl-modules

So it looks like this PR adds provider support, without changing engine support! Don't know if both libp11 engine and provider can be used in the same application. I think you could have both in the same openssl.cnf But one or the other should workin an application.

I spent the last few days seeing if this PR works or not, and it works for the case, and it looks like it could easily be used by others libp11 users to convert from engine to provider.

And it looks like using this provider with some simple additional mods to libp11 it could also support Ed25519, X25519 Ed448 and X448 too. See https://github.com/OpenSC/OpenSC/pull/3090

The changed to opensc.cnf looked like:

[providers_sect]
pkcs11 = pkcs11_sect
#default = default_sect
#legacy = legacy_sect

[pkcs11_sect]
# Check prov_front.c for supported parameters
#
# Some parameters could be overridden with environment variable value. See prov_front.c
# Like: PKCS11MODULE, PKCS11VERBOSE, PKCS11FORCELOGIN, ... 
PKCS11VERBOSE = 1 
PKCS11FORCELOGIN = 1 
#
# alternative name (if we were compiling the provider as pkcs11.so, we won't need this)
identity = pkcs11prov
# name of the libp11 pkcs11 provider
####module = @@OPENSSL_PROVIDER@@
# like MODULE_PATH in case of the libp11 pkcs11 engine
######pkcs11module = @@MODULE@@
# libp11 log level (0 - off, >0 - on) 
verbose = 1 

# force login to the PKCS#11 module
force_login = 1 

#use OpenSSL digest commands
digest_disabled = 1 

# do not use pkcs11 module  for random number
rand_disabled = 1 

And run this command where: ossl-active is ossl-3.3.1

PKCS11FORCELOGIN=1 \
FORCELOGIN=1 \
PKCS11MODULE=/opt/ossl-active/lib/opensc-pkcs11.so \
PKCS11VERBOSE=1 \
VERBOSE=1 \
OPENSSL_MODULES=/opt/ossl-active/lib/ossl-modules \
gdb --args  /opt/ossl-3.3.1/bin/openssl req -verbose -provider pkcs11prov -provider default -sha256 -new -key pkcs11://pkcs11:type=private\;id=%01 -out cards/8103.myreq.9A.pem -text