pcks11-provider doesn't work when FIPS mode is enabled

[GeorgePantelakis]

Describe the bug When we are running pcks11-provider and FIPS mode is enabled, the provider is loaded by OpenSSL but none of the functions are FIPS compatible so none of them is called or used. Trying to load a pkcs11 URI makes OpenSSL fall back to the default provider and try to open it as a file. This fails and the "Failed to open OpenSSL store: error:8000000D:system library::Permission denied" error message is printed.

To Reproduce We are using libssh API to reproduce this bug. Steps to reproduce the behavior:

1. Create an RSA key-pair with OpenSSL
2. Create softhsm tokens from the key-pair
3. Export "PKCS11_PROVIDER_MODULE=/usr/lib64/pkcs11/libsofthsm2.so"
4. Run the test harness with the pksc11 URI from the tokens
5. See error in the output

Expected behavior The pcks11 keys (both RSA and ECDSA) should be able to be loaded by the pcks11-provider in FIPS mode.

Operating environment (please complete the following information):

• OS: RedHat Enterprise Linux
• Version: 10

Token and application used (please complete the following information):

• Device: softhsm tokens
• PKCS11 Driver version: 0.5
• Application: libssh API
• Version: 0.11.1

Harness libssh API C code converted to txt. Feel free to reproduce it however you see fit. simple-libssh-client.txt

[simo5]

Isn't this basically a duplicate of #164 ?

[beldmit]

It is - but #164 was basically forgotten