sshd fails to start when provider is enabled

[sharkcz]

When a config file prepared by ibmca-provider-opensslconfig is used, sshd (and some other services) fails to start in Fedora Rawhide.

The journal contains

Jan 13 09:53:29 fedora systemd[1]: Starting sshd.service - OpenSSH server daemon...
Jan 13 09:53:29 fedora sshd[770]: PRNG is not seeded
Jan 13 09:53:29 fedora systemd[1]: sshd.service: Main process exited, code=exited, status=255/EXCEPTION
Jan 13 09:53:29 fedora systemd[1]: sshd.service: Failed with result 'exit-code'.
Jan 13 09:53:29 fedora systemd[1]: Failed to start sshd.service - OpenSSH server daemon.

The package versions are

openssl-3.0.7-2.fc38.s390x
libica-4.2.0-1.fc38.s390x
openssl-ibmca-2.3.1-3.fc38.s390x

ibmca is configured with --disable-engine --enable-provider --libdir=/usr/lib64/ossl-modules --with-libica-cex --with-libica-version=4

CCing @kkaarreell

[sharkcz]

the environment is a Rawhide VM on z14 with Fedora 36 with virt-preview repo as the KVM hypervisor.

[ifranzki]

Could this be related to https://github.com/openssl/openssl/issues/18578 ? Or is this because sshd does only allow a certain number of file descriptors to be open in its sandbox?

In general I would suggest to have a separate OpenSSL config file for specific applications, and NOT enable a provider (and/or engine) system wide. You can use the OPENSSL_CONF environment variable to point to a specific config file for the application you want to use a provider with.

[ifranzki]

Can you tell if this is due to a process fault (e.g segfault)? For me it does not look like. I would assume you should see more infos in syslog if it would be a crash.

[sharkcz]

seems there is no additional information, no segfault or coredump ...

also not related to SELinux, same issue in permissive mode

[ifranzki]

I have just tried to reproduce this by running sshd with a separate OpenSSL config file which activates the provider:

# OPENSSL_CONF=/root/openssl.cnf.ibmca-provider /usr/sbin/sshd -p 2222

Config file /root/openssl.cnf.ibmca-provider was generated with ibmca-provider-opensslconfig from the default system wide OpenSSL config file (this was on Fedora 36).

This also shows PRNG is not seeded. This is because the default system wide OpenSSL config file does NOT define the default provider, and thus no other algorithms than those provided by the IBMCA provider are available, i.e. no PRNG.

Please note the warning that ibmca-provider-opensslconfig prints:

WARNING: The IBMCA provider was added to section [provider_sect].
Make sure that you have configured and activated at least one other provider, e.g. the default provider!

Successfully generated openssl.cnf.ibmca-provider file.  Please review this configuration
and, if you are happy with the changes, replace /etc/pki/tls/openssl.cnf with
this file.

This warning is meant seriously ;-)

A similar warning is actually generated as comment inside the config file right after the definitions of the IBMCA provider in the provider section:

ibmca_provider = ibmca_provider_section
# Make sure that you have configured and activated at least one other provider!

If no other provider is defined, then the default provider will be loaded automatically. But once you define another provider you must make sure that you also define the default provider in the config file!

Uncommenting the definitions for the default provider solves the problem for me:

alg_section = evp_properties_section
[provider_sect]
default = default_sect
##legacy = legacy_sect
##
ibmca_provider = ibmca_provider_section
# Make sure that you have configured and activated at least one other provider!
[default_sect]
activate = 1
##
##[legacy_sect]
##activate = 1

[sharkcz]

So this is an documentation/configuration issue, it makes sense.

I am aware of the warning from ibmca-provider-opensslconfig, but you need to be an expert in openssl configuration to be able to review the result. Which I am not and I suspect most users won't be.

[sharkcz]

looks to me the tool creates

...
alg_section = evp_properties_section
[provider_sect]
ibmca_provider = ibmca_provider_section
1default = default_sect
##legacy = legacy_sect
##
[default_sect]
activate = 1

##[legacy_sect]
...

see the 1default =, it might be the culpript ...

[sharkcz]

after fixing it, it now lists both default and ibmca in openssl list -providers

[ifranzki]

see the 1default =, it might be the culpript ...

No, not for me:

alg_section = evp_properties_section
[provider_sect]
##default = default_sect
##legacy = legacy_sect
##
ibmca_provider = ibmca_provider_section
# Make sure that you have configured and activated at least one other provider!
##[default_sect]
##activate = 1
##
##[legacy_sect]
##activate = 1

Please note that besides the lines it adds, it takes all lines from the default OpenSSL config file as is. So if the 1default was already there, it will also be in the generated file (garbage in -> garbage out ;-) ).

[sharkcz]

I have now confirmed that in Rawhide it is the script that breaks the config by adding the "1".

https://kojipkgs.fedoraproject.org//packages/openssl/3.0.7/2.fc38/x86_64/openssl-libs-3.0.7-2.fc38.x86_64.rpm has the source openssl.cfg, which looks good

[ifranzki]

hm there were only 2 commits changing the script and I don't see that those would be fixing something like that.... And it does not happen on my system. Can you send me your script version please?

[sharkcz]

if you are on F-36, then it might be a newer perl, pcre or some other library in Rawhide.

https://kojipkgs.fedoraproject.org//packages/openssl-ibmca/2.3.1/3.fc38/s390x/openssl-ibmca-2.3.1-3.fc38.s390x.rpm is what I am using

[ifranzki]

Hm just tested the script without the 2 commits, and it seems that without https://github.com/opencryptoki/openssl-ibmca/commit/072e32bb199ff772148f1cbe0b2faadf9ab33c12 it indeed generates the 1, although in front of the ## in my case:

[provider_sect]
ibmca_provider = ibmca_provider_section
1##default = default_sect
##legacy = legacy_sect

From the change in this commit I can't tell why this happens....

Nevertheless, I would suggest that you pick the two commits (both fixing the script): https://github.com/opencryptoki/openssl-ibmca/commit/072e32bb199ff772148f1cbe0b2faadf9ab33c12 https://github.com/opencryptoki/openssl-ibmca/commit/e90203dbc9bf0d9a4488af470adf11852860991a

[sharkcz]

let me try with all the patches on top of 2.3.1 ...

[sharkcz]

I should have read rhbz#2140028 more carefully, it's all there :-)

[ifranzki]

Interestingly in F-36 the default OpenSSL config file has the default provider lines commented out. Thus on F-36 one would still need to uncomment the default provider definitions to get this to work.

On your system this seems not to be the case, and the default provider is defined by default, which should result in a working setup right away (assuming the script does not add the 1).

[sharkcz]

It's the same with F-37, likely the default config changed in openssl 3.0.7, because both F-36 and F-37 are on openssl 3.0.5. I believe we (will) have 3.0.7 in RHEL-9, so the script should prepare a working config there as well.

[kkaarreell]

I have tested this on the latest (today's) build for RHEL-9. It looks like this

# /usr/share/doc/openssl-ibmca/ibmca-provider-opensslconfig
WARNING: The IBMCA provider was added to section [provider_sect].
Make sure that you have configured and activated at least one other provider, e.g. the default provider!

Successfully generated openssl.cnf.ibmca-provider file.  Please review this configuration
and, if you are happy with the changes, replace /etc/pki/tls/openssl.cnf with
this file.
# diff openssl.cnf.ibmca-provider /etc/pki/tls/openssl.cnf
59d58
< alg_section = evp_properties_section
61,62c60
< ibmca_provider = ibmca_provider_section
< 1default = default_sect
---
> default = default_sect
397,409d394
< 
< [ibmca_provider_section]
< identity = ibmca
< module = ibmca-provider.so
< activate = 1
< #debug = yes
< #fips = yes
< #algorithms = RSA,EC,DH
< algorithms = ALL
< #fallback-properties = provider=default
< 
< [evp_properties_section]
< default_properties = ?provider=ibmca

Once I have downloaded the latest version of the script from github it works fine and sshd starts properly with the generated config.

# diff openssl.cnf.ibmca-provider /etc/pki/tls/openssl.cnf
59d58
< alg_section = evp_properties_section
64,65d62
< ibmca_provider = ibmca_provider_section
< # Make sure that you have configured and activated at least one other provider!
398,410d394
< 
< [ibmca_provider_section]
< identity = ibmca
< module = ibmca-provider.so
< activate = 1
< #debug = yes
< #fips = yes
< #algorithms = RSA,EC,DH
< algorithms = ALL
< #fallback-properties = provider=default
< 
< [evp_properties_section]
< default_properties = ?provider=ibmca

[ifranzki]

Thanks @kkaarreell. I am closing this issue now, since the problems is now fully understood and already fixed.

[sharkcz]

ack and thanks for your support