Pkcs11 engine causes crash in OpenSSL 1.1.0b in RSA_public_decrypt.

[nased0]

Hello! I have compiled libp11 in VS2015 for OpenSSL 1.1.0b by modifying OPENSSL_LIB in make.rules.mak . Unfortunately engine pkcs11.dll crashes OpenSSL 1.1, because it does not set RSA method "rsa_pub_dec" - rsa->meth->rsa_pub_dec is a NULL. pointer! In contrary, RSA method rsa_priv_dec points to to "pkcs11.dll ! pkcs11_rsa_priv_dec_method()". Please correct this error, I really need working pkcs11 engine for my application, and I'm fed up with memory leaks in OpenSSL 1.0.2!

[nased0]

OpenSSL crash when I execute command: openssl s_client -msg -cert its1_crt.pem -key its1_key.pem -CAfile cck.pem -no_tls1_2 -connect 172.25.223.7:443 (this IP is behind VPN). My openssl.cnf is: openssl.txt As you can see, I have engine pkcs11.dll enabled, but in this openssl command I don't even use pkcs#11. Of course it crashes when I actually use engine too! openssl s_client -engine pkcs11 -msg -tls1_1 -CAfile cck.pem -connect 172.25.223.7:443

[dengert]

In libp11/src/p11_rsa.c should line 432: ops = RSA_meth_new("libp11 RSA method", 0); be something like: ops = RSA_meth_dup(RSA_get_default_method()); RSA_meth_set1_name(ops, "libp11 RSA method"); RSA_meth_set_flags(ops, 0);

(Have not tried this.)

[nased0]

Thanks! For now I can tell that with this patch engine pkcs11.dll works with the first command I mentioned (with PEM certs), I will test the second command at work, when I have access to the card reader.

[mouse07410]

Oops! The following test was done with OpenSSL-1.0.2j. But the proposed patch should not break OpenSSL 1.0.2, right?

After applying the proposed patch on Mac OS X:

/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
FAIL: rsa-testpkcs11.softhsm
FAIL: rsa-testfork.softhsm
PASS: rsa-testlistkeys.softhsm
FAIL: rsa-evp-sign.softhsm
PASS: ec-testfork.softhsm
============================================================================
Testsuite summary for libp11 0.4.3_git
============================================================================
# TOTAL: 5
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

Content of tests/test-suite.log:

.. contents:: :depth: 2

FAIL: rsa-testpkcs11.softhsm
============================

Current directory: /Users/uri/src/libp11/tests
Source directory: .
Output directory: output.58809
-n * Initializing smart card... 
ok
Using slot 0 with a present token (0x2acd1731)
Using slot 0 with a present token (0x2acd1731)
Using slot 0 with a present token (0x2acd1731)
***************
Listing objects
***************
Using slot 0 with a present token (0x2acd1731)
Public Key Object; RSA 2048 bits
  label:      server-key
  ID:         01020304
  Usage:      encrypt, verify, wrap
Certificate Object, type = X.509 cert
  label:      server-key
  ID:         01020304
Private Key Object; RSA 
  label:      server-key
  ID:         01020304
  Usage:      decrypt, sign, unwrap
dyld: lazy symbol binding failed: Symbol not found: _RSA_meth_dup
  Referenced from: /Users/uri/src/libp11/src/.libs/libp11.2.dylib
  Expected in: flat namespace

dyld: Symbol not found: _RSA_meth_dup
  Referenced from: /Users/uri/src/libp11/src/.libs/libp11.2.dylib
  Expected in: flat namespace

./rsa-testpkcs11.softhsm: line 25: 58820 Trace/BPT trap: 5       ../examples/auth $ADDITIONAL_PARAM $PIN
Basic PKCS #11 test test failed
FAIL rsa-testpkcs11.softhsm (exit status: 1)

FAIL: rsa-testfork.softhsm
==========================

Current directory: /Users/uri/src/libp11/tests
Source directory: .
Output directory: output.58840
-n * Initializing smart card... 
ok
Using slot 0 with a present token (0x401cc042)
Using slot 0 with a present token (0x401cc042)
Using slot 0 with a present token (0x401cc042)
***************
Listing objects
***************
Using slot 0 with a present token (0x401cc042)
Private Key Object; RSA 
  label:      server-key
  ID:         01020304
  Usage:      decrypt, sign, unwrap
Certificate Object, type = X.509 cert
  label:      server-key
  ID:         01020304
Public Key Object; RSA 2048 bits
  label:      server-key
  ID:         01020304
  Usage:      encrypt, verify, wrap
dyld: lazy symbol binding failed: Symbol not found: _RSA_meth_dup
  Referenced from: /Users/uri/src/libp11/src/.libs/libp11.2.dylib
  Expected in: flat namespace

dyld: Symbol not found: _RSA_meth_dup
  Referenced from: /Users/uri/src/libp11/src/.libs/libp11.2.dylib
  Expected in: flat namespace

Child terminated by signal #5
Slot manufacturer......: SoftHSM project
Slot description.......: SoftHSM slot ID 0x401cc042
Slot token label.......: libp11-test
Slot token manufacturer: SoftHSM project
Slot token model.......: SoftHSM v2
Slot token serialnr....: 7f3e7196401cc042
Slot manufacturer......: SoftHSM project
Slot description.......: SoftHSM slot ID 0x401cc042
Slot token label.......: libp11-test
Slot token manufacturer: SoftHSM project
Slot token model.......: SoftHSM v2
Slot token serialnr....: 7f3e7196401cc042
Slot manufacturer......: SoftHSM project
Slot description.......: SoftHSM slot ID 0x401cc042
Slot token label.......: libp11-test
Slot token manufacturer: SoftHSM project
Slot token model.......: SoftHSM v2
Slot token serialnr....: 7f3e7196401cc042
Slot manufacturer......: SoftHSM project
Slot description.......: SoftHSM slot ID 0x401cc042
Slot token label.......: libp11-test
Slot token manufacturer: SoftHSM project
Slot token model.......: SoftHSM v2
Slot token serialnr....: 7f3e7196401cc042
FAIL rsa-testfork.softhsm (exit status: 1)

FAIL: rsa-evp-sign.softhsm
==========================

Current directory: /Users/uri/src/libp11/tests
Source directory: .
Output directory: output.58909
-n * Initializing smart card... 
ok
Using slot 0 with a present token (0x2e0cccd9)
Using slot 0 with a present token (0x2e0cccd9)
Using slot 0 with a present token (0x2e0cccd9)
***************
Listing objects
***************
Using slot 0 with a present token (0x2e0cccd9)
Private Key Object; RSA 
  label:      server-key
  ID:         01020304
  Usage:      decrypt, sign, unwrap
Public Key Object; RSA 2048 bits
  label:      server-key
  ID:         01020304
  Usage:      encrypt, verify, wrap
Certificate Object, type = X.509 cert
  label:      server-key
  ID:         01020304
At main.c:193:
- SSL error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library: dso_dlfcn.c:187
- SSL error:25070067:DSO support routines:DSO_load:could not load the shared library: dso_lib.c:233
- SSL error:260B6084:engine routines:DYNAMIC_LOAD:dso not found: eng_dyn.c:467
- SSL error:2606A074:engine routines:ENGINE_by_id:no such engine: eng_list.c:390
Basic PKCS #11 test, using ctrl failed
FAIL rsa-evp-sign.softhsm (exit status: 1)

[nased0]

I confirm that engine pkcs11 for OpenSSL 1.1.0b works correctly with PKCS#11 card after this patch. Unfortunately there are numerous memory leaks. Unpublished version 0.4.3 (pulled from this page) works with OpenSSL 1.1 and card reader and has much less memleaks. libp11.zip

[nased0]

By comparing libp11-0.4.2 with the attached working version 0.4.3 I have determined that this regression is caused by following entry in src/eng_front.c, function: static int bind_helper(ENGINE *e)

[nased0]

In the newest master, to make it work I have commented one line in eng_front.c

/* This internal function is used by ENGINE_pkcs11() and possibly by the
 * "dynamic" ENGINE support too */
static int bind_helper(ENGINE *e)
{
    if (!ENGINE_set_id(e, PKCS11_ENGINE_ID) ||
            !ENGINE_set_destroy_function(e, engine_destroy) ||
            !ENGINE_set_init_function(e, engine_init) ||
            !ENGINE_set_finish_function(e, engine_finish) ||
            !ENGINE_set_ctrl_function(e, engine_ctrl) ||
            !ENGINE_set_cmd_defns(e, engine_cmd_defns) ||
            !ENGINE_set_name(e, PKCS11_ENGINE_NAME) ||
#ifndef OPENSSL_NO_RSA
//          !ENGINE_set_RSA(e, PKCS11_get_rsa_method()) ||
#endif
            !ENGINE_set_load_pubkey_function(e, load_pubkey) ||
            !ENGINE_set_load_privkey_function(e, load_privkey)) {
        return 0;
    } else {
        return 1;
    }
}

[dwmw2]

That's the "correct fix", but unfortunately we need to register something to work around bugs in OpenSSL 1.0.2's openssl command line tool.

[nased0]

But I'm using OpenSSL 1.1.0b!

[dwmw2]

Yeah, sorry, that's broken too.

[nased0]

I don't know the problem with openssl command line tool, commands that I cited execute correctly when I remove this "workaround". But this workaround made pkcs11 engine completely unusable for practical purposes, like communication with https SOAP server using curl and PKCS#11 cards. If there is a problem with openssl command line tool, there is openssl group on github too!

[dwmw2]

cf. https://github.com/openssl/openssl/pull/1639 and https://github.com/openssl/openssl/pull/1642 and https://github.com/openssl/openssl/pull/1643 and probably one or two more.

[nased0]

Fine. RSA_private_decrypt and RSA_private_encrypt are apparently not enough for OpenSSL 1.1. If you want to set pkcs11 engine as RSA provider, then you must implement and register functions RSA_public_decrypt and RSA_public_encrypt, used for signing and verifying messages.

But you should be wary of limitations of PKCS#11 cards, for example one of PKCS#11 cards I use (Encard Enigma 4.0, based on Gemalto cards) cannot encrypt (or sign) messages of length 83 bytes, therefore TLS 1.2 can't work in OpenSSL 1.0.2, but works in .Net implementation. But other card I use (Unizeto cryptoCertum 3.0) does not have this problem

And the card can be easily removed from the reader, so it is not a good idea to set it as a default RSA provider, which is active even for operations, that don't require the card (i.e. my first cited openssl command).

[dengert]

Can't sign the 83 byte message? Is the card or PKCS#11 implementation limited in someway? What mechanisms does it support? The hash and maybe the padding may need to be done in software then the card can do the RSA. Do you have any type of trace?

[mouse07410]

A normal RSA card shouldn't be able to sign a message whose size is not equal to the key size. Your application should pad the message before passing it to the card.

Your Unizeto is rather an exception than a rule.

[nased0]

@dengert I'm still talking about pkcs11 engine - function pkcs11_private_encrypt Its call to function: rv = CRYPTOKI_call(ctx, C_Sign(spriv->session, (CK_BYTE *)from, flen, to, &size)); where flen = 83, size = 128 returns rv = 33 = CKR_DATA_LEN_RANGE (plaintext input has bad length) Which results in OpenSSL error: error:80009021:Vendor defined:PKCS11_rsa_encrypt:Data len range It happens when I try to connect using TLS 1.2, therefore I have to use less secure TLS 1.1 with this Gemalto card.

[dengert]

In src/p11rsa.c the logic looks strange, and the comments indicate there are issues: 99 /* Try signing first, as applications are more likely to use it / 100 rv = CRYPTOKI_call(ctx, 101 C_SignInit(spriv->session, &mechanism, kpriv->object)); 102 if (!rv) 103 rv = pkcs11_authenticate(key); 104 if (!rv) 105 rv = CRYPTOKI_call(ctx, 106 C_Sign(spriv->session, (CK_BYTE )from, flen, to, &size)); 107 if (rv == CKR_KEY_FUNCTION_NOTPERMITTED) { 108 / OpenSSL may use it for encryption rather than signing / 109 rv = CRYPTOKI_call(ctx, 110 C_EncryptInit(spriv->session, &mechanism, kpriv->object)); 111 if (!rv) 112 rv = pkcs11_authenticate(key); 113 if (!rv) 114 rv = CRYPTOKI_call(ctx, 115 C_Encrypt(spriv->session, (CK_BYTE *)from, flen, to, &size));

It looks like it is trying signing first which is failing, with a different return code, and it never tries the encryption.

Can you do any debugging?
Using OpenSC-spy would help a lot, as it would show the PKCS#11 calls and returns.
Spy can be used to with other vendor's PKCS#11 modules too.

Especially the mechanism being requested. There are 17 RSA mechanisms that do combinations of hashing, padding and RSA operations. (CKM_RSA_X_509 is the RAW RSA) It may be that the PKCS#11 module or the card does not expose CKM_RSA_X_509 for C_Sign to be for a specific hash and is testing for it. I have seen this with older cards that only worked with SHA1. Where as C_Encrypt does not.

[dengert]

Actually, why is C_SignInit and C_Sign being called at all? The caller should be passing the data to encrypt or the hash OID and hash to encrypt. Based on the size, the caller is expecting PKCS#11 can do the padding.

I would try and delete lines 99 to 108 and line 116 "}" and see it it works for you.

[mouse07410]

@dengert, are you sure you're looking at the correct version? In my libp11 master the relevant code looks like:

99       /* Try signing first, as applications are more likely to use it */
100        rv = CRYPTOKI_call(ctx,
                C_SignInit(spriv->session, &mechanism, kpriv->object));
        if (rv == CKR_USER_NOT_LOGGED_IN)
                rv = pkcs11_authenticate(key);
        if (!rv)
                rv = CRYPTOKI_call(ctx,
                        C_Sign(spriv->session, (CK_BYTE *)from, flen, to, &size));
        if (rv == CKR_KEY_FUNCTION_NOT_PERMITTED) {
                /* OpenSSL may use it for encryption rather than signing */
                rv = CRYPTOKI_call(ctx,
                        C_EncryptInit(spriv->session, &mechanism, kpriv->object));
                if (rv == CKR_USER_NOT_LOGGED_IN)
                        rv = pkcs11_authenticate(key);
                if (!rv)
                        rv = CRYPTOKI_call(ctx,
                                C_Encrypt(spriv->session, (CK_BYTE *)from, flen, to, &size));
        }

which appears perfectly correct to me:

1. C_SignInit() is done
2. If it returns CKR_USER_NOT_LOGGED_IN,pkcs11_authenticate() is called
3. If the return call from either C_SignInit() or pkcs11_authenticate() is CKR_OK, C_Sign() is called, otherwise
4. If C_SignInit() returned CKR_KEY_FUNCTION_NOT_PERMITTED, the code attempts to follow the C_EncryptInit() path.

What problem do you see there?

And I strongly second the suggestion to look at padding.

[dengert]

I sounds like a PKCS#11 implementation for the card he has that does not know how to do a sign 83 bytes. Need more debugging info.

I don't think it is a padding issue, as pkcs11_mechanism(&mechanism, padding) is called and sets the mechanism to CKM_RSA_PKCS (have PKCS#11 or card) do the padding, CKM_RSA_X_509, (padding already done) or CKM_RSA_X9_31 based on the padding parameter or pkcs11_private_encrypt.

Still need more debug info.

[nased0]

I have of course patched eng_front.c by removing !ENGINE_set_RSA(e, PKCS11_get_rsa_method()) || P.S. kpriv->object = 4

[dwmw2]

And the card can be easily removed from the reader, so it is not a good idea to set it as a default RSA provider, which is active even for operations, that don't require the card (i.e. my first cited openssl command).

Right. There is a problem here with the OpenSSL engine API, and bugs in the users.

We shouldn't need to call ENGINE_register_RSA(). That registers an RSA method which can be used as the default. At least it does if applications wantonly call ENGINE_set_default()... which they shouldn't be doing if all they want to do is use a key from the engine, but they do. Including the openssl command line tool.

Worse, the openssl command line tool has refcounting bugs which mean it will free the engine and then continue to use it... unless it happens to expose a default method for something, which will "coincidentally" pin it in place and stop the use-after-free bug from biting.

So we have to set a default method, and basically we have to make it pass through to the software internal methods when it's used on a key which is not PKCS#11. So this isn't about making PKCS#11 tokens the default RSA provider even for software stuff... it's just about hackish workarounds for OpenSSL problems.

The third reason to provide a method is because (at least in OpenSSL 1.1) you need to do that in order to get correct use counts working on the keys provided from your engine... so that it all works correctly when people do things like

e = ENGINE_init();
k = ENGINE_get_private_key(e, ...)l
ENGINE_finish(e);
r = EVP_PKEY_get1_RSA(k);
EVP_PKEY_free(k);
/* At thist point the engine still exists */
RSA_Sign(r, ...);
RSA_free(r);
/* And *now* the engine can be unloaded */

But right now that last one is entirely theoretical, because our internal use counts on keys are entirely hosed, and we end up with recursive references and nothing could ever be freed. And in fact even for OpenSSL 1.1 (where the ->engine fields are in opaque structs and we can't set them behind OpenSSL's back) there are other workarounds we can do to make the refcounting work so we still don't need to use ENGINE_set_RSA() for any purpose except working around that openssl(1) use-after-free bug.

But for OpenSSL: <= 1.1 if we want to support being invoked from openssl(1) then we do need to provide that default.

[nased0]

Look, I don't care about which openssl command works with this engine and which don't (except the simple ones I cited). I do care that this engine works with curl like this: curl --engine pkcs11 --tlsv1.1 --key-type ENG --cert-type ENG -k https://172.25.223.7:443/cepik/api/skp?wsdl It is the way my application works. And it doesn't work if this hack is enabled!

[dwmw2]

Yeah, that's a curl bug. IT shouldn't be calling ENGINE_set_RSA() either. https://github.com/curl/curl/pull/1042

Although I thought @mtrojnar had made the pass-through to internal software methods work...

[nased0]

But it doesn't explain why commands openssl s_client -engine pkcs11 -msg -tls1_1 -CAfile cert\cck.pem -connect 172.25.223.7:443 and openssl s_client -msg -cert cert\its1_crt.pem -key cert\its1_key.pem -CAfile cert\cck.pem -no_tls1_2 -connect 172.25.223.7:443 crash when this hack is enabled, and execute correctly when disabled.

[dwmw2]

The curl bug doesn't. The other explanation above does.

[dwmw2]

Sorry, perhaps not explicitly enough to warrant such a terse reply; on my phone while putting kids to bed. The problem is with the pass through to the software implementations, when opens sleep or curl wrongly call ENGINE_set_default ()

[dwmw2]

OpenSSL or curl ffs. Not "opens sleep"

[dwmw2]

Maybe we should provide an MD5 implementation to work around this issue, not an RSA method.

[nased0]

https://www.openssl.org/docs/man1.0.1/crypto/engine.html

ENGINE _e; const char engine_id = "ACME"; ENGINE_load_builtin_engines(); e = ENGINE_by_id(engineid); if(!e) / the engine isn't available _/ return; if(!ENGINEinit(e)) { / the engine couldn't initialise, release 'e' _/ ENGINE_free(e); return; } if(!ENGINE_set_defaultRSA(e)) / This should only happen when 'e' can't initialise, but the previous * statement suggests it did. _/ abort(); ENGINE_set_default_DSA(e); ENGINE_set_defaultciphers(e); / Release the functional reference from ENGINEinit() / ENGINEfinish(e); / Release the structural reference from ENGINE_by_id() / ENGINE_free(e);

[nased0]

ENGINE_finish releases only functional reference ENGINE_free releases structural reference and deallocates memory when the last structural reference is released.

[dwmw2]

Yes. Why?

[nased0]

Simply look at the source code of theses functions ENGINE_new, ENGINE_init, ENGINE_finish and ENGINE_free. There are two completely separate reference counters, called functional and structural.

[dwmw2]

Yes. Why? (are you telling me this)

[nased0]

Because I don't understand why you have pasted this e = ENGINE_init(); k = ENGINE_get_private_key(e, ...)l ENGINE_finish(e); r = EVP_PKEY_get1_RSA(k); EVP_PKEYfree(k); /* At thist point the engine still exists / RSA_Sign(r, ...); RSAfree(r); / And now the engine can be unloaded */

I can't find function ENGINE_get_private_key in google or opennsl docs, so I can't assume what it does to the reference counters.

[dwmw2]

Ah right, yes there was a missing ENGINE_free(). Let me try that again:

e = ENGINE_by_id("pkcs11");
ENGINE_init(e);
ENGINE_free(e);
k = ENGINE_get_private_key(e, ...)l
ENGINE_finish(e);
r = EVP_PKEY_get1_RSA(k);
EVP_PKEY_free(k);
/* At this point the engine still exists */
RSA_sign(r, ...);
RSA_free(r);
/* And *now* the engine has beene unloaded */

Good spot, but it doesn't change anything. Our refcounting is still utterly hosed for the above use case. And openssl(1) still does this:

e = ENGINE_by_id("pkcs11");
ENGINE_set_default(e); /* Does nothing if the engine doesn't provide default software methods */
ENGINE_free(e); /* bye bye engine */
ENGINE_get_private_key(e); /* BOOM! */

[nased0]

Then it is openssl(1) that needs to be corrected, not pkcs11 engine.

[nased0]

Why are you calling ENGINE_finish after ENGINE_free? IMO functional reference set by ENGINE_init prevents ENGINE_free from unloading the engine, but I will have to check its source...

[dwmw2]

Yes, openssl(1) should be fixed. In 1.1 and 1.0.2 both. Hence all those OpenSSL tickets I referred to earlier

Yes, the functional reference which is held from ENGINE_init() and then again implicitly in the RSA object in the EVP_PKEY object that you get back from ENGINE_get_private_key() will prevent the engine from being unloaded. Or it would if our internal refcounting wasn't entirely hosed.

[dengert]

In regard to the 83 bytes. The trace shows mechanism= 1 = CKM_RSA_PKCS Based on the size of (to) = 128 bytes this is a 1024 bit key. So the 83 bytes are to be padded by the PKCS#11 module. It should handle 117 bytes.

Why the CKR_DATA_LEN_RANGE is returned is not clear. This looks like an issue with the "ENCARD PKCS#11" module

[nased0]

I checked the code of engine_unlocked_init and engine_free_util. engine_free_util (called by ENGINE_free) does not check functional references at all, but engine_unlocked_init (called by ENGINE_init) does this funny thing: e->struct_ref++; e->funct_ref++; And ENGINE_finish calls engine_unlocked_finish, which calls engine_free_util after decreasing the functional reference. So it is a structural reference increased by ENGINE_init that prevents ENGINE_free from unloading the engine.

[dwmw2]

@nased0, yes, that's all fairly much as described. A functional reference implies a structural reference too. So if you take a functional reference and then free all the explicit structural references, then the engine will be unloaded when that final functional reference goes away — be it an explicit functional reference you free with ENGINE_finish(), or an implicit functional reference that is supposed to be tied to the key objects the engine gives out, which goes away when the last key is destroyed.

[nased0]

@dengert Yes, I'm aware that the Encard cards have this problem and the pkcs11 engine should do the padding for them.

[dengert]

OK, so are you saying the Encard PKCS#11 module does not support CKM_RSA_PKCS? If so, CKR_DATA_LEN_RANGE is not the correct return code.

But it could also be that libp11 is assuming that all PKCS#11 modules support CKM_RSA_PKCS. I don't see a call to C_GetMechanismList in libp11.

If you have time, can you use the opensc-spy as the PKCS#11 module, and set in registry: HKLM\Software\OpenSC Project\PKCS11-Spy\Module to point at the Encard PKCS#11 module. See:

https://github.com/OpenSC/OpenSC/wiki/Using-OpenSC

If you have the OpenSC pkcs11-tool.exe on Windows, You could also use it with --module and -M to see what mechanisms it does support.

[mouse07410]

I confess I don't fully understand what you are trying to accomplish:

Signed (data len=83, signature 256 bytes): rv=1
18 7d e5 03 e7 83 cb 3c 3a 9c 3e a9 42 e7 ff d2 
. . . . .
63 9e ab f5 bc e8 a7 b2 e8 6e 77 59 2f 47 ae 12 

Signature verified successfully: rv=1

Above was done with OpenSSL-1.0.2j, current master libp11, YubiKey NEO (that does not do PKCS1 on the card), and EVP_PKEY_Sign() group of calls. Verification was with EVP_PKEY_verify() group. Observe that OpenSSL did the padding appropriately.

[nased0]

I just want to connect with https server protected by these cards using TLS 1.2. I can do it using .Net implementation. It is OpenSSL that wants client to encrypt 83 bytes as part of the negotiations.

[dengert]

The NEO does not do the padding in hardware, but the OpenSC PKCS#11 does it in software in src/libopensc/padding.c

[mouse07410]

The NEO does not do the padding in hardware,

Yes, precisely. And AFAIK, neither does CAC.

but the OpenSC PKCS#11 does it in software in src/libopensc/padding.c

I see. I thought that in my case the padding was done by OpenSSL (I was setting EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PADDING);), but I could be wrong, and OpenSSL could "outsource" the work to OpenSC library.

[dengert]

Yes the hashing(if needed), padding and RSA operations need to be done in that order and can be done at different levels . In @nased0 case using libp11 and engine, It appears the Encard PKCS#11 module is asked to do the padding and RSA because mechanisum == CKM_RSA_PKCS. But it appears the Encard PKCS#11 can not do CKM_RSA_PKCS. It also appears that libp11 is assuming all PKCS#11 modules support CKM_RSA_PKCS. A robust libp11 would check what mechanisums are supported and if CKM_RSA_PKCS is not and CKM_RSA_X_509 is, then libp11 would do the padding and then pass the padded data to the PKCS#11 using mechanisum = CKM_RSA_X_509.

The OpenSC PKCS#11 can do some padding operations including CKM_RSA_PKCS and it if the card supports it, to will let the card do the padding and RSA on the card as one operation. If not the padding will be done in software, and the padded data then passed to the card to do a RSA RAW operation.

[mouse07410]

...It appears the Encard PKCS#11 module is asked to do the padding and RSA because mechanisum == CKM_RSA_PKCS. But it appears the Encard PKCS#11 can not do CKM_RSA_PKCS.

You seem to have nailed the problem.

Since my example shows that OpenSSL/libp11/OpenSC can do all the padding operations in software, the solution seems to be to somehow convince whatever entity (libp11?) that tries to offload PKCS1 padding to the card, to do it in software instead.

It also appears that libp11 is assuming all PKCS#11 modules support CKM_RSA_PKCS.

I'm not sure I understand or agree. If the above were true, I shouldn't be able to get libp11 to work with YubiKey (or CAC) at all, because neither of these devices supports CKM_RSA_PKCS. Yet libp11 never tried to push that operation to the card, always doing the padding in software.

I conjecture that it's something that Encard PKCS#11 library tells libp11.

A robust libp11 would check what mechanisums are supported and if CKM_RSA_PKCS is not and CKM_RSA_X_509 is, then libp11 would do the padding and then pass the padded data to the PKCS#11 using mechanisum = CKM_RSA_X_509.

Yes.

But if libp11 does not do that check - why in my case OpenSSL+libp11 correctly understands that padding must be done in software?

In case it matters:

$ opensc-tool --version
OpenSC-OpenSC_working_snapshot-8-gae738ea, rev: ae738ea, commit-time: 2016-10-13 17:06:58 -0400
$ opensc-tool -l
# Detected readers (pcsc)
Nr.  Card  Features  Name
0    Yes             Yubico Yubikey NEO OTP+U2F+CCID
$ pkcs11-tool -M
Using slot 0 with a present token (0x0)
Supported mechanisms:
  SHA-1, digest
  SHA256, digest
  SHA384, digest
  SHA512, digest
  MD5, digest
  RIPEMD160, digest
  GOSTR3411, digest
  ECDSA, keySize={256,256}, hw, sign, other flags=0x1800000
  ECDH1-COFACTOR-DERIVE, keySize={256,256}, hw, derive, other flags=0x1800000
  ECDH1-DERIVE, keySize={256,256}, hw, derive, other flags=0x1800000
  RSA-X-509, keySize={1024,3072}, hw, decrypt, sign, verify
  RSA-PKCS, keySize={1024,3072}, hw, decrypt, sign, verify
  SHA1-RSA-PKCS, keySize={1024,3072}, sign, verify
  SHA256-RSA-PKCS, keySize={1024,3072}, sign, verify
  SHA384-RSA-PKCS, keySize={1024,3072}, sign, verify
  SHA512-RSA-PKCS, keySize={1024,3072}, sign, verify
  MD5-RSA-PKCS, keySize={1024,3072}, sign, verify
  RIPEMD160-RSA-PKCS, keySize={1024,3072}, sign, verify
$ 

Needless to say, most of the above capabilities are not supported by the token itself, and have to be handled by the software (so the list appears to me more about what pkcs11-tool can do rather than what the token supports directly).