engine_pkcs11 leaks memory

[oerdnj]

Hey, I've implemented a custom memory tracking in BIND 9 for the external libraries, including OpenSSL, and it found that engine_pkcs11 module is leaking following memory (lines should match OpenSSL 3.0.5):

    ptr 0x56056e775b50 size 264 file ../crypto/evp/pmeth_lib.c line 130
    ptr 0x56056e708570 size 120 file ../crypto/ec/ec_kmeth.c line 184
    ptr 0x56056e708310 size 120 file ../crypto/rsa/rsa_meth.c line 48
    ptr 0x56056e708280 size 24 file ../crypto/rsa/rsa_meth.c line 71
    ptr 0x56056e774c10 size 264 file ../crypto/evp/pmeth_lib.c line 130
    ptr 0x56056e6f6020 size 56 file ../crypto/threads_pthread.c line 50

It's following methods:

1. ptr 0x55dd947bb8f0 size 24 file ../crypto/rsa/rsa_meth.c line 71 --> RSA_meth_set1_name()
2. ptr 0x55dd947bbbe0 size 120 file ../crypto/ec/ec_kmeth.c line 184 --> EC_KEY_METHOD_new()
3. ptr 0x55dd947bb980 size 120 file ../crypto/rsa/rsa_meth.c line 48 --> RSA_meth_dup()
4. ptr 0x55dd948287b0 size 264 file ../crypto/evp/pmeth_lib.c line 130 --> EVP_PKEY_meth_new()

The call tree can be found in the attached PDF: jeprof.pdf

This simple code ENGINE_free(ENGINE_by_id("pkcs11")); is enough to reproduce the issue.

Looking at the code, it seems like the code that would free the method allocated in PKCS11_get_rsa_method() (and others) was never actually implemented:

/* This function is *not* currently exported */
void PKCS11_rsa_method_free(void)
{
        free_rsa_ex_index();
}

[dengert]

Ssee https://github.com/OpenSC/libp11/pull/470#issuecomment-1219913973 and make sure the free callback is also added. Further comments discuses OpenSSL 1.1 and OpenSSL 3.0 while engines are still available

471 tries to address the dup problem.

[mtrojnar]

I merged the commits from #471. Is this issue fixed now?