OpenSSL 3 can't parse an RSA private key

[nponsard]

I tried with OpenSSL version 3.0.9 / 3.1.1 / 3.2.0-dev. Libp11 version 0.4.12 and (compiled from source) commit 74497e0fa5b69b15790d6697e1ebce13af842d4c

When you execute the command

openssl pkey -engine pkcs11 -inform ENGINE -in "pkcs11:model=PKCS%2315%20emulated [...]

You get the error from OpenSSL :

40D72375427F0000:error:068000DE:asn1 encoding routines:asn1_template_ex_i2d:illegal zero content:crypto/asn1/tasn_enc.c:374:
40D72375427F0000:error:020C0100:rsa routines:rsa_priv_encode:malloc failure:crypto/rsa/rsa_ameth.c:154:
40D72375427F0000:error:03000092:digital envelope routines:EVP_PKEY2PKCS8:private key encode error:crypto/evp/evp_pkey.c:161:
40D72375427F0000:error:04800073:PEM routines:do_pk8pkey:error converting private key:crypto/pem/pem_pk8.c:133:

This command works with the older Openssl 1.1 and libp11 0.4.9.

I did some debugging and found out that it tries to read the d value of the RSA key. In p11_rsa.c, function pkcs11_get_rsa : the pointer to d is set to NULL in the call to RSA_set0_key.

I tried to create a pointer to a big number with the value 0, I still had the same problem but with the p factor. After setting values for RSA_set0_factors and RSA_set0_crt_params :

    int size = 1;
    CK_BYTE data = 0;
    BIGNUM *bnd = NULL;
    bnd = BN_bin2bn(&data, size, bnd);

    RSA_set0_key(rsa, rsa_n, rsa_e, bnd);

    BIGNUM *bnp = NULL;
    bnp = BN_bin2bn(&data, size, bnp);
    BIGNUM *bnq = NULL;
    bnq = BN_bin2bn(&data, size, bnq);

    RSA_set0_factors(rsa, bnp, bnq);

    BIGNUM *bndmp1 = NULL;
    bndmp1 = BN_bin2bn(&data, size, bndmp1);
    BIGNUM *bndmq1 = NULL;
    bndmq1 = BN_bin2bn(&data, size, bndmq1);
    BIGNUM *bniqmp = NULL;
    bniqmp = BN_bin2bn(&data, size, bniqmp);

    RSA_set0_crt_params(rsa, bndmp1, bndmq1, bniqmp);

The OpenSSL command worked.

[jans23]

@frankmorgner might have an idea.

[dengert]

It maybe related to RSA_set0_key being deprecated in OpenSSL 3.0 and internally OpenSSL does not handle a NULL "d" See: https://www.openssl.org/docs/man3.1/man3/RSA_set0_key.html

In libp11 and PKCS11 there are private keys and public keys. In OpenSSL there are just keys. Libp11 looks like it is writing the public key components into an OpenSSL RSA key. "d" is the private secret.

The OpenSSL error messages don't seem to be related to the problem.

You did some debugging... Do you have an backtrace or other info?

It looks like you are using OpenSC to some smart card too. What card? Are libp11 and OpenSC linked to the same version of OpenSSL? Many cards will never give you the RSA "d".

Maybe libp11 should be be updated to not use the deprecated RSA calls.

[nponsard]

It maybe related to RSA_set0_key being deprecated in OpenSSL 3.0 and internally OpenSSL does not handle a NULL "d" See: https://www.openssl.org/docs/man3.1/man3/RSA_set0_key.html

It looks like that could be the problem, in the man page it says that only n and e must be non NULL but apparently they expect everything to be non-NULL ?

The OpenSSL error messages don't seem to be related to the problem.

You did some debugging... Do you have an backtrace or other info?

It happens when OpenSSL tries to convert the EVP_PKEY that libp11 created into a PKCS8 structure with the function EVP_PKEY2PKCS8. The errors happens in the call of pkey->ameth->priv_encode that is implemented by rsa_priv_encode in rsa_ameth the OpenSSL code. It then gets into a loop in ASN1_item_ex_i2d in tasn_enc.c where it reads the templates (I did not find where it's defined) and one of the "templates" needs to read the d value.

It looks like you are using OpenSC to some smart card too. What card? Are libp11 and OpenSC linked to the same version of OpenSSL? Many cards will never give you the RSA "d". I tested with the Nitrokey 3 and OpenSC. I tried with libp11 and OpenSC linked to the same OpenSSL version (compiled locally) and the problem is still here.

After reading the man page it looks like this is more a bug in OpenSSL, but since this function is being deprecated I don't know if I should report the issue. Also the whole "engine" interface of OpenSSL is being deprecated in favor of the "provider api" https://github.com/openssl/openssl/blob/master/README-ENGINES.md .

[nponsard]

I think I found the root of my problem, it was user error and confusion. Since the security keys don't give access to the private components of the key I only wanted to get the public key. Turns out I only needed to add the -pubin or -pubout option to the OpenSSL.

[dengert]

Great. In regard to OpenSSL 3.0 and providers vs engines, you may want to look at https://github.com/latchset/pkcs11-provider

[nponsard]

Thanks !