Supporting FIPS mode in FCOS

[darkmuggle]

When booting with fips=1 as a karg, boot will fail. Under FIPS, any key-material must be generated with FIPS enabled, otherwise the key material (SSH keys, crypto, etc) is considered to be plain-text.

The failure comes in from https://github.com/dracutdevs/dracut/blob/049/modules.d/01fips/fips.sh#L111-L135 where the HMAC is checked.

In order to ensure FIPS compliance, COS should generate the hmac before the dracut-prepivot.service runs.

[darkmuggle]

/cc @jlebon @cgwalters @imcleod @miabbott

[cgwalters]

We're generating the HMAC server side today - this is just about client-side verification right? And wasn't the problem more that that code didn't know how to find the ostree-managed kernels?

So in other words, I think the bug here is "dracut FIPS module can't find kernel", not anything related to key generation and FIPS. Right?

[jlebon]

We're generating the HMAC server side today - this is just about client-side verification right? And wasn't the problem more that that code didn't know how to find the ostree-managed kernels?

I was looking at a patch to just have libostree install the HMAC file alongside the kernel image in /boot if it finds one. This should match what happens in traditional RHEL/Fedora.

So in other words, I think the bug here is "dracut FIPS module can't find kernel", not anything related to key generation and FIPS. Right?

I think the logic that's there should work for us, but there seems to be a bug that's confusing things. Here is the warning/error message:

[    7.815544] dracut: FATAL: FIPS integrity test failed
[    7.815545] dracut-pre-pivot[1025]: Warning: /boot/.vmlinuz-4.18.0-146.el8.x86_64.hmac does not exist
[    7.820628] dracut: Refusing to continue
[    7.815547] dracut-pre-pivot[1025]: Warning: /boot/.vmlinuz-4.18.0-146.el8.x86_64.hmac does not exist

That specifically comes from:

        BOOT_IMAGE_HMAC="/boot/${BOOT_IMAGE_PATH}.${BOOT_IMAGE_NAME}.hmac"
        if ! [ -e "${BOOT_IMAGE_HMAC}" ]; then
            warn "${BOOT_IMAGE_HMAC} does not exist"
            return 1
        fi

but the BOOT_IMAGE_PATH is being reset to "" here so it's not looking for the HMAC alongside the kernel image. I think there's a bug in that code; I'm playing right now with:

diff --git a/modules.d/01fips/fips.sh b/modules.d/01fips/fips.sh
index c6de3083..b256b11b 100755
--- a/modules.d/01fips/fips.sh
+++ b/modules.d/01fips/fips.sh
@@ -118,7 +118,7 @@ do_fips()

         if [ -z "$BOOT_IMAGE_NAME" ]; then
             BOOT_IMAGE_NAME="vmlinuz-${KERNEL}"
-        elif ! [ -e "/boot/${BOOT_IMAGE_PATH}/${BOOT_IMAGE}" ]; then
+        elif ! [ -e "/boot/${BOOT_IMAGE_PATH}/${BOOT_IMAGE_NAME}" ]; then
             #if /boot is not a separate partition BOOT_IMAGE might start with /boot
             BOOT_IMAGE_PATH=${BOOT_IMAGE_PATH#"/boot"}
             #on some achitectures BOOT_IMAGE does not contain path to kernel

With that + libostree installing the HMAC file, I think this should work. (And traditional RHEL/Fedora will keep working via falling back to /boot/.vmlinuz-*.hmac).

[cgwalters]

I was looking at a patch to just have libostree install the HMAC file alongside the kernel image in /boot if it finds one. This should match what happens in traditional RHEL/Fedora.

Oh right yes, we clearly need that.

[jlebon]

OK so with the following patches:

• https://github.com/ostreedev/ostree/pull/1962
• https://github.com/coreos/rpm-ostree/pull/1934
• https://github.com/dracutdevs/dracut/pull/661

I can boot RHCOS and FCOS fine in FIPS mode! :tada:

[root@coreos ~]# cat /proc/sys/crypto/fips_enabled
1

[fifofonix]

+1 vote for above to have this addressed.

[bgilbert]

@fifofonix Can you elaborate on your use case? I wouldn't expect FIPS to be on the radar of a lot of FCOS users.

[JaimeMagiera]

+1 as well.

My use case is that the large organization I work for (https://www.icpsr.umich.edu) wants to use OKD, and FCOS, for developing and hosting web applications. Our contracts often require FIPS compliance.

Let me know if you need any more specific details.

[fifofonix]

I also work for a large .edu and we're eager to do everything we can to build secure software. To the extent that there are additional security benefits by enabling FIPS mode in the kernel, over-and-above the application of the FIPS crypto policy (which we are doing), we would like to benefit from these.

[fifofonix]

Good news I think. As part of speculative testing for fedora37 test day I revisited setting FIPS and it now seems to work which is awesome. I executed fips-mode-setup --enable --no-bootcfg within the systemd unit applying the crypto policies and then subsequently manually modified kernel arguments with rpm-ostree kargs --append="fips=1".

# cat /proc/sys/crypto/fips_enabled
1

And examining the last boot journald:

$ sudo journalctl -b | grep -E 'FIPS|fips'
Sep 29 14:24:30 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-52c15085333e022e7f5d11adcff4da848f4547a641dd5210fbdc04aa503260c5/vmlinuz-5.19.9-300.fc37.x86_64 mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/fedora-coreos/52c15085333e022e7f5d11adcff4da848f4547a641dd5210fbdc04aa503260c5/0 ignition.platform.id=vmware root=UUID=2a1ba988-bc1e-4fbb-b156-43ee4fe9527c rw rootflags=prjquota boot=UUID=f1424889-a126-4560-abd2-c6b75c06efd2 fips=1
Sep 29 14:24:30 localhost kernel: Kernel command line: BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-52c15085333e022e7f5d11adcff4da848f4547a641dd5210fbdc04aa503260c5/vmlinuz-5.19.9-300.fc37.x86_64 mitigations=auto,nosmt console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.1/fedora-coreos/52c15085333e022e7f5d11adcff4da848f4547a641dd5210fbdc04aa503260c5/0 ignition.platform.id=vmware root=UUID=2a1ba988-bc1e-4fbb-b156-43ee4fe9527c rw rootflags=prjquota boot=UUID=f1424889-a126-4560-abd2-c6b75c06efd2 fips=1
Sep 29 14:24:30 localhost kernel: fips mode: enabled
Sep 29 14:24:30 localhost kernel: alg: poly1305 (poly1305-simd) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: xchacha12 (xchacha12-simd) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: xchacha20 (xchacha20-simd) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: chacha20 (chacha20-simd) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: ecdh-nist-p192 (ecdh-nist-p192-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: blake2b-512 (blake2b-512-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: blake2b-384 (blake2b-384-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: blake2b-256 (blake2b-256-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: blake2b-160 (blake2b-160-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: md5 (md5-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: digest_null (digest_null-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: compress_null (compress_null-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: ecdsa-nist-p384 (ecdsa-nist-p384-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: ecdsa-nist-p256 (ecdsa-nist-p256-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: ecdsa-nist-p192 (ecdsa-nist-p192-generic) is disabled due to FIPS
Sep 29 14:24:30 localhost kernel: alg: dh (dh-generic) is disabled due to FIPS

So, we can close this issue?

[jlebon]

Good to know that it works when manually enabling it.

So, we can close this issue?

Before we do that, we should discuss if we want to properly support FIPS mode in FCOS. If we do, we should add CI for it and make the UX for enabling it easier. See https://github.com/coreos/ignition/issues/1323 for the latest discussions around that.

[fifofonix]

Better UX would be good but in the mean time I think the following works if you want to enable/apply FIPS auto-triggering a reboot with added required kernel argument.

Example given also applies some other minor ACME-specific crypto hardening. It will get re-triggered each time there is a major fedora version increment...to benefit from any ongoing improvements to the FIPS-defined baselines.

variant: fcos
version: 1.4.0
storage:
  files:
    - path: /etc/crypto-policies/policies/modules/ACME.pmod
      mode: 0644
      contents:
        inline: |
          mac = -HMAC-SHA1
          cipher = -AES-256-CBC -AES-128-CBC
          #cipher@openssl = -AES-256-CBC -AES-128-CBC
          #cipher@openssh-server = -AES-256-CBC -AES-128-CBC
    # SELinux policies written in cil format can be installed with
    # semodule command.  We do this with systemd unit.
systemd:
  units:
    # Apply FIPS:ACME crypto settings after every OS update...and reboot...
    - name: acme-set-crypto-policy.service
      enabled: true
      contents: |
        [Unit]
        Description=Enable FIPS:ACME crypto settings
        After=network-online.target
        Before=zincati.service
        ConditionPathExists=!/var/lib/%N.%w.stamp

        [Service]
        Type=oneshot
        ExecStart=/bin/sh -e -c 'FEDORA_VERSION=$(cat /etc/os-release | grep VERSION_ID | cut -d\= -f2); \
                /bin/rm -rf /var/lib/%N.*.stamp; \
                /bin/podman run --privileged -i --rm -v /etc/crypto-policies:/etc/crypto-policies \
                    registry.fedoraproject.org/fedora:$FEDORA_VERSION /bin/bash -c \
                        "dnf install -y crypto-policies-scripts && update-crypto-policies \
                            --no-reload --set FIPS:ACME && fips-mode-setup --enable --no-bootcfg"; \
                if [ $(cat /proc/sys/crypto/fips_enabled) == 0 ]; then \
                    rpm-ostree kargs --append="fips=1"; \
                    /bin/touch /var/lib/%N.%w.stamp; \
                    reboot; \
                else \
                    /bin/touch /var/lib/%N.%w.stamp; \
                fi'
        RemainAfterExit=yes

        [Install]
        WantedBy=multi-user.target

[dustymabe]

We discussed this at the community meeting today.

13:13:27        dustymabe | #agreed adding a fips=1 kernel argument via Ignition 
                          | becomes the canonical low level API to signal desire
                          | for FIPS mode, FCOS will add glue to do the remaining 
                          | userspace portions on firstboot, and we will add a
                          | minimal kola test that verifies this works; no butane 
                          | sugar yet, and documentation that briefly describes
                          | this state

[travier]

Note that adding FIPS support to FCOS does not make FCOS (or more precisely the openssl build inside) FIPS certified. So this is mostly useless (from the conformance point of view) as it's unlikely that Fedora's openssl build will ever be FIPS certified. We have the same issue in CentOS Stream (CoreOS).

[jdoss]

I believe it's no longer useless with the changes that are coming to the OpenSSL RPM in F38. If you check out the spec there are patches added that conditionally enable FIPS in OpenSSL if fips= is set in the kernel.

I was able to get FIPS fully working with CoreOS Layering:

Containerfile

# Copy and enable FIPS things
COPY fcos/files/ACME.pmod /etc/crypto-policies/policies/modules/ACME.pmod
RUN update-crypto-policies --no-reload --set FIPS:ACME && fips-mode-setup --enable --no-bootcfg
# This can get removed once Fedora 38 hits stable in FCOS
RUN rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/openssl/3.0.8/1.fc38/x86_64/openssl-libs-3.0.8-1.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/openssl/3.0.8/1.fc38/x86_64/openssl-3.0.8-1.fc38.x86_64.rpm

Butane

- name: fcos-rebase.service
  enabled: true
  contents: |
    [Unit]
    Description=Rebase FCOS to Container Image
    ConditionPathExists=!/var/lib/fcos-rebase.stamp
    After=network-online.target coreos-ignition-firstboot-complete.service coreos-update-ca-trust.service
    Wants=network-online.target

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    Restart=on-failure
    RestartSec=10s
    ExecStart=rpm-ostree rebase --bypass-driver --experimental ostree-unverified-registry:{{ container_registry }}/{{ container_name }}:{{ container_tag }}
    ExecStartPost=/bin/touch /var/lib/fcos-rebase.stamp
    ExecStartPost=rpm-ostree kargs --append="fips=1"
    ExecStartPost=systemctl reboot
    [Install]
    WantedBy=basic.target

Testing FIPS enabled in OpenSSL

[root@localhost ~]# rpm -qa |grep openssl
openssl-libs-3.0.8-1.fc38.x86_64
openssl-3.0.8-1.fc38.x86_64
[root@appliance ~]# openssl md5 foo 
Error setting digest
006E833A567F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (MD5 : 97), Properties ()
006E833A567F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:crypto/evp/digest.c:254:
[root@localhost ~]# openssl sha1 foo 
SHA1(foo)= 2dbe8ab0745f09dd67bfe2acabdc79583283bdd0```

[Malix-Labs]

Is it fixed ?