softhsm2 leaks memory

[oerdnj]

Hey, I've implemented a custom memory tracking in BIND 9 for the external libraries, including OpenSSL, and it found that softhsm2 module is leaking following memory (lines should match OpenSSL 3.0.5):

    ptr 0x56056e775b50 size 264 file ../crypto/evp/pmeth_lib.c line 130
    ptr 0x56056e708570 size 120 file ../crypto/ec/ec_kmeth.c line 184
    ptr 0x56056e708310 size 120 file ../crypto/rsa/rsa_meth.c line 48
    ptr 0x56056e708280 size 24 file ../crypto/rsa/rsa_meth.c line 71
    ptr 0x56056e774c10 size 264 file ../crypto/evp/pmeth_lib.c line 130
    ptr 0x56056e6f6020 size 56 file ../crypto/threads_pthread.c line 50

It's following methods:

1. ptr 0x55dd947bb8f0 size 24 file ../crypto/rsa/rsa_meth.c line 71 --> RSA_meth_set1_name()
2. ptr 0x55dd947bbbe0 size 120 file ../crypto/ec/ec_kmeth.c line 184 --> EC_KEY_METHOD_new()
3. ptr 0x55dd947bb980 size 120 file ../crypto/rsa/rsa_meth.c line 48 --> RSA_meth_dup()
4. ptr 0x55dd948287b0 size 264 file ../crypto/evp/pmeth_lib.c line 130 --> EVP_PKEY_meth_new()

[oerdnj]

The leak can be triggered just by loading and freeing engine, e.g. ENGINE_free(ENGINE_by_id('pkcs11')); will trigger the memory leak...

[oerdnj]

Thinking about it a little more, it could also be a bug in engine_pkcs11.

[oerdnj]

Which is in fact true...

[oerdnj]

Reported here, if anyone is interested: https://github.com/OpenSC/libp11/issues/475

[rijswijk]

Thanks for the write-up @oerdnj (and finding the place where the leak is)