Bind helper methods in engine_ctrl() only for OpenSSL versions older than 3.0

[olszomal]

With OpenSSL 3.x, an engine might be used if defined in openssl.cnf. This causes memory leaks when engine control commands are set in the config file. For OpenSSL 3.0 and newer, add engine routines only after loading keys.

Fixed #358

[dengert]

Have you looked at #540? The engine support on OpenSSL is on its way out. #540 looks good, but needs a rebase by the author. It looks like you could define both a provider and an engine but may not both be usable from same application.

[olszomal]

@dengert Thank you for the invitation. I’m preparing to take on this challenge.

[mtrojnar]

Have you looked at #540? The engine support on OpenSSL is on its way out. #540 looks good, but needs a rebase by the author. It looks like you could define both a provider and an engine but may not both be usable from same application.

Could you explain why they may not both be usable? The libp11 API was designed to allow for more than one context at a time. I can't find any fundamental technical reasons that could prevent that.

[dengert]

Could you explain why they may not both be usable? No, but with all the the issues of cleaning up, including #359 and and #540 I as a user want to stick with only engine or provider in the same application.

[mtrojnar]

No, but with all the the issues of cleaning up, including #359 and and #540 I as a user want to stick with only engine or provider in the same application.

Picking one interface and using libp11 either directly or over the engine or over the future provider in accordance with the KISS principle is a good idea. It's not a requirement though.

359 mostly frustrated Valgrind users with OCD symptoms. Over the years, I only saw one report of a user whose in-house application was compulsively removing and re-adding the engine for each signature, where the leak caused an actual issue. A workaround seemed fairly straightforward though: to stop removing the engine after each signature.