FIPS kernel is installed but not correctly set up on s390x

[cpaelzer]

Hi, on my nonx86 tests I've had many systems not come back up at all (other to be debugged problem), but the one that did come up again has not booted into the FIPS kernel.

Enabling went fine

root@server20:~# sudo ua enable fips
sudo: unable to resolve host server20
One moment, checking your subscription first
Installation of additional packages are required to make this system FIPS
compliant.
Are you sure? (y/N) y
Updating package lists
Installing FIPS packages
FIPS enabled
A reboot is required to complete install.
root@server20:~# reboot

Packages are installed

root@server20:~# dpkg -l | grep fips
ii  fips-initramfs                        0.0.5.2                                    s390x        FIPS 140-2 kernel tests
ii  libssl1.0.0:s390x                     1.0.2g-1ubuntu4.fips.4.6.3                 s390x        Secure Sockets Layer toolkit - shared libraries
ii  libssl1.0.0-hmac:s390x                1.0.2g-1ubuntu4.fips.4.6.3                 s390x        Secure Sockets Layer toolkit - FIPS HMAC integrity check files
ii  linux-fips                            4.4.0.1002.3                               s390x        Complete Linux kernel for FIPS systems.
ii  linux-fips-headers-4.4.0-1002         4.4.0-1002.2                               all          Header files related to Linux kernel version 4.4.0
ii  linux-headers-4.4.0-1002-fips         4.4.0-1002.2                               s390x        Linux kernel headers for version 4.4.0 on System 390x SMP
ii  linux-headers-fips                    4.4.0.1002.3                               s390x        Linux kernel headers for FIPS systems.
ii  linux-image-4.4.0-1002-fips           4.4.0-1002.2                               s390x        Linux kernel image for version 4.4.0 on System 390x SMP
ii  linux-image-extra-4.4.0-1002-fips     4.4.0-1002.2                               s390x        Linux kernel extra modules for version 4.4.0 on System 390x SMP
ii  linux-image-fips                      4.4.0.1002.3                               s390x        Linux kernel image for FIPS systems.
ii  linux-image-hmac-4.4.0-1002-fips      4.4.0-1002.2                               s390x        HMAC file for linux kernel image for version 4.4.0 on System 390x SMP
ii  openssh-client                        1:7.2p2-4ubuntu2.fips.2.10.1               s390x        secure shell (SSH) client, for secure access to remote machines
ii  openssh-client-hmac:s390x             1:7.2p2-4ubuntu2.fips.2.10.1               s390x        FIPS HMAC integrity check files for secure shell (SSH) client.
ii  openssh-server                        1:7.2p2-4ubuntu2.fips.2.10.1               s390x        secure shell (SSH) server, for secure access from remote machines
ii  openssh-server-hmac:s390x             1:7.2p2-4ubuntu2.fips.2.10.1               s390x        FIPS HMAC integrity check files for secure shell (SSH) server.
ii  openssh-sftp-server                   1:7.2p2-4ubuntu2.fips.2.10.1               s390x        secure shell (SSH) sftp server module, for SFTP access from remote machines
ii  openssl                               1.0.2g-1ubuntu4.fips.4.6.3                 s390x        Secure Sockets Layer toolkit - cryptographic utility
ii  ubuntu-fips                           1.0.2                                      s390x        Install and configure linux-fips kernel and user space modules

Kernel is present:

root@server20:~# ll /boot/vmlinuz*
lrwxrwxrwx 1 root root       25 Apr 29 06:04 /boot/vmlinuz -> vmlinuz-4.4.0-210-generic
-rw------- 1 root root 12968704 Dec  4  2015 /boot/vmlinuz-4.3.0-2-generic
-rw------- 1 root root  3789968 Apr 27  2017 /boot/vmlinuz-4.4.0-1002-fips
-rw------- 1 root root 13530624 Mar  9  2016 /boot/vmlinuz-4.4.0-12-generic
-rw------- 1 root root  3891432 Apr 16 10:07 /boot/vmlinuz-4.4.0-210-generic
-rw------- 1 root root  3698800 Oct 20  2016 /boot/vmlinuz-4.4.0-46-generic

But my zipl boot menu is unchanged:

root@server20:~# cat /etc/zipl.conf
[defaultboot]
defaultmenu = menu1

[ubuntu]
target      = /boot
image=/boot/vmlinuz
ramdisk=/boot/initrd.img
#parameters="root=/dev/mapper/vg_ubuntu-root crashkernel=196M"
parameters="root=/dev/mapper/vg_ubuntu-root crashkernel=196M fips=1 bootdev=LABEL=UBOOT"

[ubuntu-new]
target      = /boot
image=/boot/vmlinuz-4.4.0-12-generic
ramdisk=/boot/initrd.img-4.4.0-12-generic
#parameters="root=/dev/mapper/vg_ubuntu-root crashkernel=196M"
parameters="root=/dev/mapper/vg_ubuntu-root crashkernel=196M fips=1 bootdev=LABEL=UBOOT"

:menu1
target      = /boot
1           = ubuntu
2           = ubuntu-new
default     = 1
prompt      = 1
timeout     = 10

And since the update neither modified zipl.conf NOR did overwrite the kernel/initrd symlinks

root@server20:~# ll /boot/vmlinuz /boot/initrd.img
lrwxrwxrwx 1 root root 28 Apr 29 06:03 /boot/initrd.img -> initrd.img-4.4.0-210-generic
lrwxrwxrwx 1 root root 25 Apr 29 06:04 /boot/vmlinuz -> vmlinuz-4.4.0-210-generic

I'm still booted into a non-fips kernel

root@server20:~# uname -a
Linux server20 4.4.0-210-generic #242-Ubuntu SMP Fri Apr 16 09:57:28 UTC 2021 s390x s390x s390x GNU/Linux

[cpaelzer]

Just FYI - in two different systems the upgrade worked. There the symlinks are updated AND zipl.conf is modified.

ubuntu@xenial-ua-scsi:~$ uname -a
Linux xenial-ua-scsi 4.4.0-1002-fips #2-Ubuntu SMP Thu Apr 27 19:35:14 UTC 2017 s390x s390x s390x GNU/Linux
ubuntu@xenial-ua-scsi:~$ cat /etc/zipl.conf
# This has been modified by the cloud image build process
[defaultboot]
default=ubuntu

[ubuntu]
target = /boot
image = /boot/vmlinuz
ramdisk = /boot/initrd.img
#parameters = root=LABEL=manual-rootfs
parameters = root=LABEL=manual-rootfs fips=1
ubuntu@xenial-ua-scsi:~$ ll /boot/
total 37360
drwxr-xr-x  3 root root     4096 Apr 29 07:22 ./
drwxr-xr-x 22 root root     4096 Apr 29 07:35 ../
-rw-r--r--  1 root root   537007 Apr 27  2017 abi-4.4.0-1002-fips
-rw-------  1 root root    13824 Apr 29 07:22 bootmap
-rw-r--r--  1 root root    65307 Apr 27  2017 config-4.4.0-1002-fips
-rw-r--r--  1 root root    65666 Apr 16 10:07 config-4.4.0-210-generic
drwxr-xr-x  2 root root     4096 Apr 29 07:22 grub/
lrwxrwxrwx  1 root root       26 Apr 29 07:21 initrd.img -> initrd.img-4.4.0-1002-fips
-rw-r--r--  1 root root 12502226 Apr 29 07:22 initrd.img-4.4.0-1002-fips
-rw-r--r--  1 root root 12194648 Apr 29 07:22 initrd.img-4.4.0-210-generic
lrwxrwxrwx  1 root root       28 Apr 22 22:47 initrd.img.old -> initrd.img-4.4.0-210-generic
-rw-------  1 root root  2567485 Apr 27  2017 System.map-4.4.0-1002-fips
-rw-------  1 root root  2588452 Apr 16 10:07 System.map-4.4.0-210-generic
lrwxrwxrwx  1 root root       23 Apr 29 07:21 vmlinuz -> vmlinuz-4.4.0-1002-fips
-rw-------  1 root root  3789968 Apr 27  2017 vmlinuz-4.4.0-1002-fips
-r--------  1 root root      160 Apr 27  2017 .vmlinuz-4.4.0-1002-fips.hmac
-rw-------  1 root root  3891432 Apr 16 10:07 vmlinuz-4.4.0-210-generic
lrwxrwxrwx  1 root root       25 Apr 22 22:47 vmlinuz.old -> vmlinuz-4.4.0-210-generic

So maybe all this fix will be about is detecting custom setups (like my error case zipl menu) and gracefully telling the user that UA couldn't set it up due to custom config and that the user has to configure the bootlader himself to enable kernel /boot/...

[orndorffgrant]

Tracking in SC-635

[cpaelzer]

Re-tried - it really only breaks on custom configs which isn't UAs fault. Please close this.

But it seems the fips kernel on s390x is not booting (at least on my xenial test).