provider {rsa,ec,dh}key tests fail on z14

[sharkcz]

I am getting test failures for the {rsa,ec,dh}key tests fail on z14 with this build configuration

CFLAGS= -O2 -Wall
IBMCA engine:      yes
  default library: libica.so.4
IBMCA provider:    yes
  libica library:  libica-cex

The system is Fedora 36 with

libica-4.0.1-1.fc36.s390x
openssl-3.0.2-5.fc36.s390x

test-suite.log is here

=======================================================
   openssl-ibmca 2.3.0: test/provider/test-suite.log
=======================================================

# TOTAL: 13
# PASS:  10
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 3

.. contents:: :depth: 2

ERROR: rsakey
=============

Context is not using the IBMCA provider, but 'default'
Failure for RSA-512
Context is not using the IBMCA provider, but 'default'
Failure for RSA-1024
Context is not using the IBMCA provider, but 'default'
Failure for RSA-2048
Context is not using the IBMCA provider, but 'default'
Failure for RSA-4096
Context is not using the IBMCA provider, but 'default'
Failure for RSA-PSS-512
Context is not using the IBMCA provider, but 'default'
Failure for RSA-PSS-1024
Context is not using the IBMCA provider, but 'default'
Failure for RSA-PSS-2048
Context is not using the IBMCA provider, but 'default'
Failure for RSA-PSS-4096
ERROR rsakey (exit status: 99)

ERROR: eckey
============

Context is not using the IBMCA provider, but 'default'
Failure for NID_X9_62_prime192v1
Context is not using the IBMCA provider, but 'default'
Failure for NID_secp224r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_X9_62_prime256v1
Context is not using the IBMCA provider, but 'default'
Failure for NID_secp384r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_secp521r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP160r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP192r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP224r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP256r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP320r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP384r1
Context is not using the IBMCA provider, but 'default'
Failure for NID_brainpoolP512r1
ERROR eckey (exit status: 99)

ERROR: dhkey
============

Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe2048 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe2048 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe3072 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe3072 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe4096 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe4096 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe6144 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe6144 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe8192 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_ffdhe8192 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_1536 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_1536 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_2048 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_2048 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_3072 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_3072 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_4096 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_4096 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_6144 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_6144 (DHX)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_8192 (DH)
Context is not using the IBMCA provider, but 'default'
Failure for NID_modp_8192 (DHX)
ERROR dhkey (exit status: 99)

[ifranzki]

This is most likely the same as https://github.com/openssl/openssl/issues/18262 . Has been fixed with https://github.com/openssl/openssl/pull/18269 . Commits: https://github.com/openssl/openssl/commit/4b1b629725970384d6cf4dafe9e83e54859574cd https://github.com/openssl/openssl/commit/70dc0b6d27a11a7f64fe914a3f376988ad1b1720

[sharkcz]

Thanks for the pointers. Is it right that I don't see the problem with z15?

[ifranzki]

This should not be HW dependent, the OpenSSL bug simply does not call our provider anymore at all.

[sharkcz]

Interesting, because with the same openssl package (again F-36), but on z15 with CEX7, all the tests are passing.

[ifranzki]

Well, could it be that your z14 does not have any crypto adapters ?

IBMCA depends on what libica-cex supports, and it registers only for those algorithms that libica-cex reports to be supported in hardware (you can check with icainfo-cex).

If none of the algorithms are supported by libica-cex, then the provider does not register for any of them, and the test results would be what you see. It would use the default provider for all those algorithms, and not the IBMCA provider.

We intentionally don't silently skip these tests when the IBMCA provider did not register for that algorithms, because there are other reasons why the provider could not be used (for example the OpenSSL bug mentioned above). So it only makes sense to run the IBMCA provider tests on a system that has crypto adapters supporting the algorithms.

[sharkcz]

yes, the z14 is without crypto adapters

Our guidelines say we are expected to run test suites during the build, but we can't ensure the builders will have crypto adapters. I believe the user needs to know (or be notified) the failures are expected on systems without crypto adapters. Ideally such tests should be skipped. I think the engine is already skipping some tests if some condition isn't met. Or is it tests in libica?

[ifranzki]

The thing is that testing the provider without crypto adapter is completely useless, since it would not be invoked in such cases. So you would simply not testing anything.

[ifranzki]

I might have found a way to skip the tests in such cases, but still keep the check for using the right provider if libica supports the algorithms, see https://github.com/opencryptoki/openssl-ibmca/pull/85.

Would be nice if you could give the PR a test in your z14 environment and check if it produces the expected result (i.e. skip of those 3 test cases).