search

linuxboot / heads

A minimal Linux that runs as a coreboot or LinuxBoot ROM payload to provide a secure, flexible boot environment for laptops, workstations and servers.
https://osresearch.net/
GNU General Public License v2.0
1.42k stars 186 forks source link

LUKS encrypted NixOS 23.05+ graphical installer deploys plain luks unlock secret under unnencrypted /boot's initramfs : CVE-2023-36476) #1348

Closed septem9er closed 1 month ago

septem9er commented 1 year ago

Please identify some basic details to help process the report

A. Provide Hardware Details

Librem 14 1. What board are you using (see list of boards here)? Librem 14 2. Does your computer have a dGPU or is it iGPU-only?

3. Who installed Heads on this computer?

4. What PGP key is being used?

5. Are you using the PGP key to provide HOTP verification?

B. Identify how the board was flashed

1. Is this problem related to updating heads or flashing it for the first time?

2. If the problem is related to an update, how did you attempt to apply the update?

3. How was Heads initially flashed

4. Was the board flashed with a maximized or non-maximized/legacy rom?

5. If Heads was externally flashed, was IFD unlocked?

C. Identify the rom related to this bug report

1. Did you download or build the rom at issue in this bug report?

2. If you downloaded your rom, where did you get it from?

Please provide the release number or otherwise identify the rom downloaded According to system info: FW_VER: PureBoot-Release-24 Kernel: Linux 5.10.5-PureBoot 3. If you built your rom, which repository:branch did you use?

4. What version of coreboot did you use in building?

5. In building the rom where did you get the blobs?

Please describe the problem

Describe the bug When installing NixOS with LUKS encrypted root partition (boot is unencrypted), the system does not boot. When installing nix unencrypted it does work.

I did not fill out some of the information above, as the Librem 14 was preinstalled with coreboot and heads by purism. So I do cannot answer some of those questions.

If I can provide any additional information or help identify the issue, I'll gladly do that.

To Reproduce

Expected behavior The system should be able to boot NixOS Actual behavior Nothing happens after starting the new kernel Screenshots Boot process: boot process NOTE: I forced boot without tamper protection for the boot in this screenshot. Current partition layout: partition layout

Additional context In the matrix chat one of you told me they would like to see the kexec line vs the line in the grub.cfg. So here is the full /boot/grub/grub.cfg

# Automatically generated.  DO NOT EDIT THIS FILE!

        search --set=drive1 --fs-uuid 8B97-CA37
        if [ -s $prefix/grubenv ]; then
          load_env
        fi

        # ‘grub-reboot’ sets a one-time saved entry, which we process here and
        # then delete.
        if [ "${next_entry}" ]; then
          set default="${next_entry}"
          set next_entry=
          save_env next_entry
          set timeout=1
          set boot_once=true
        else
          set default=0
          set timeout=5
        fi

        function savedefault {
            if [ -z "${boot_once}"]; then
            saved_entry="${chosen}"
            save_env saved_entry
            fi
        }

        # Setup the graphics stack for bios and efi systems
        if [ "${grub_platform}" = "efi" ]; then
          insmod efi_gop
          insmod efi_uga
        else
          insmod vbe
        fi

            insmod font
            if loadfont ($drive1)//converted-font.pf2; then
              insmod gfxterm
              if [ "${grub_platform}" = "efi" ]; then
                set gfxmode=auto
                set gfxpayload=keep
              else
                set gfxmode=1024x768
                set gfxpayload=text
              fi
              terminal_output gfxterm
            fi

            background_color '#2F302F'

            insmod png
            if background_image --mode 'normal' ($drive1)//background.png; then
              set color_normal=white/black
              set color_highlight=black/white
            else
              set menu_color_normal=cyan/blue
              set menu_color_highlight=white/blue
            fi

menuentry "NixOS - Default" --class nixos --unrestricted {
search --set=drive1 --fs-uuid 8B97-CA37
  linux ($drive1)//kernels/z6vmhagxd308pffxnx47pvi27pbi5y5q-linux-5.15.102-bzImage init=/nix/store/y0l8j0n3g5v9zyxw04lrxhxq2qk8np2h-nixos-system-nixos-22.11.3184.9b8e5abb183/init loglevel=4
  initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd-secrets
}

submenu "NixOS - All configurations" --class submenu {
menuentry "NixOS - Configuration 1 (2023-03-21 - 22.11.3184.9b8e5abb183)" --class nixos {
search --set=drive1 --fs-uuid 8B97-CA37
  linux ($drive1)//kernels/z6vmhagxd308pffxnx47pvi27pbi5y5q-linux-5.15.102-bzImage init=/nix/store/y0l8j0n3g5v9zyxw04lrxhxq2qk8np2h-nixos-system-nixos-22.11.3184.9b8e5abb183/init loglevel=4
  initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd-secrets
}

}
tlaurion commented 1 year ago

I915 doesn't seem to be compiled in initrd.

septem9er commented 1 year ago

I915 doesn't seem to be compiled in initrd.

I did have a similiar issue when installing arch, where adding i915 to the initramfs solved the issue.

So I thought this might be the issue and already tried the following:

JonathonHall-Purism commented 1 year ago

You can list the contents of your initrd and see if i915 is in there. Something like:

zcat <your-initrd> | cpio -t | grep i915

septem9er commented 1 year ago

Thanks. Seems like I added the i915 module succesfully to the initrd. I checked the contents and there are paths for i915 now.

However, the boot still doesn't work even though the new initrd is used (it was in the kexec line).

tlaurion commented 1 year ago

Something else is at play with 5.x kernel config. I was able to see difference in behavior between 4.x and 5.x under other issue where even without i915 driver not under OS initrd, things worked before but not anymore.

Unfortunately, that will need a bit more troubleshooting

tlaurion commented 1 year ago

Note that we are trying to figure out right now what is happening with vgaarb, fbdev and i915 from 5.x kernel kexec'ing into something else. For some reason it works from 4.1x but not 5.x, and 4.x is able to boot into Tinycore (other opened issue) without Tinycore providing i915. So.... No idea for the moment, troubleshooted past two days, now going back to reading doc.

septem9er commented 1 year ago

I don't now a lot about how heads works, but is there anything I can do to help resolve this issue?

tlaurion commented 1 year ago

@septem9er originally said in OP

Describe the bug When installing NixOS with LUKS encrypted root partition (boot is unencrypted), the system does not boot. When installing nix unencrypted it does work.

@septem9er What is the output of zcat <your-initrd> | cpio -t | grep i915

tlaurion commented 1 year ago

It would be nice to provide in the issue the exact replication steps. I'm willing to install NixOS under qemu-coreboot-fbwhiptail-tpm1 to reproduce, but I would also desire to not have to search all bits of information needed to replicate your exact installation.

Can you please detail exactly how you did it so others can replicate and then we can find the issue and fix it?

tlaurion commented 1 year ago 

menuentry "NixOS - Default" --class nixos --unrestricted {
search --set=drive1 --fs-uuid 8B97-CA37
  linux ($drive1)//kernels/z6vmhagxd308pffxnx47pvi27pbi5y5q-linux-5.15.102-bzImage init=/nix/store/y0l8j0n3g5v9zyxw04lrxhxq2qk8np2h-nixos-system-nixos-22.11.3184.9b8e5abb183/init loglevel=4
  initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd-secrets
}

submenu "NixOS - All configurations" --class submenu {
menuentry "NixOS - Configuration 1 (2023-03-21 - 22.11.3184.9b8e5abb183)" --class nixos {
search --set=drive1 --fs-uuid 8B97-CA37
  linux ($drive1)//kernels/z6vmhagxd308pffxnx47pvi27pbi5y5q-linux-5.15.102-bzImage init=/nix/store/y0l8j0n3g5v9zyxw04lrxhxq2qk8np2h-nixos-system-nixos-22.11.3184.9b8e5abb183/init loglevel=4
  initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd ($drive1)//kernels/nv544pzpzldq0vpgajmgjkfym448nha4-initrd-linux-5.15.102-initrd-secrets
}

That won't fly under Heads. There is no logic to be able to replace $drive1 with 8B97-CA37. Its just removed.

@septem9er : please provide kexec call picture on boot. From my understanding init is searched on mounted / (root) and kernel and initrd are expected to be found per fs-uuid replacement through grub normally.

JonathonHall-Purism commented 1 year ago

@septem9er The issue with i915 framebuffer boot is resolved in Heads now and will be in PureBoot Release-27 (I'm working on building and testing this right now).

From the snippets above, it looks like the kernel is indeed on the boot filesystem (8B97-CA37 is a FAT32 filesystem ID, btrfs would have an actual UUID), so Heads/PB might be able to load this kernel/initrd correctly just by ignoring drive1. Maybe i915 is the only issue.

Could you please try Release-27 as soon as it is out to see if this will work with encrypted root for you?

JonathonHall-Purism commented 1 year ago

@septem9er PureBoot Release-27 is out, could you please update and give it a try? (https://source.puri.sm/firmware/utility/#usage)

septem9er commented 1 year ago

@JonathonHall-Purism After installing the Release-27, and installing NixOS with an encrypted root, the boot still fails. But in another way than before, so it did help! However, I'm not sure if heads is the issue here. Could you have a look?

That is how I installed (and partitioned) NixOS: IMG_20230623_215619_433

And this is how the boot fails because of a kernel panic: IMG_20230623_222252_320

septem9er commented 1 year ago

That won't fly under Heads. There is no logic to be able to replace $drive1 with 8B97-CA37. Its just removed.

@septem9er : please provide kexec call picture on boot. From my understanding init is searched on mounted / (root) and kernel and initrd are expected to be found per fs-uuid replacement through grub normally.

@tlaurion Ah, okay. So I guess this is the issue now. Here is the current grub.cfg:

# Automatically generated.  DO NOT EDIT THIS FILE! 

    search --set=drive1 --fs-uuid d6fa6ca3-71f6-4a65-8521-927494eded8c 
    if [ -s $prefix/grubenv ]; then 
      load_env 
    fi 

    # ‘grub-reboot’ sets a one-time saved entry, which we process here and 
    # then delete. 
    if [ "${next_entry}" ]; then 
      set default="${next_entry}" 
      set next_entry= 
      save_env next_entry 
      set timeout=1 
      set boot_once=true 
    else 
      set default=0 
      set timeout=5 
    fi 

    function savedefault { 
        if [ -z "${boot_once}"]; then 
        saved_entry="${chosen}" 
        save_env saved_entry 
        fi 
    } 

    # Setup the graphics stack for bios and efi systems 
    if [ "${grub_platform}" = "efi" ]; then 
      insmod efi_gop 
      insmod efi_uga 
    else 
      insmod vbe 
    fi 

        insmod font 
        if loadfont ($drive1)//converted-font.pf2; then 
          insmod gfxterm 
          if [ "${grub_platform}" = "efi" ]; then 
            set gfxmode=auto 
            set gfxpayload=keep 
          else 
            set gfxmode=1024x768 
            set gfxpayload=text 
          fi 
          terminal_output gfxterm 
        fi 

        background_color '#2F302F' 

        insmod png 
        if background_image --mode 'normal' ($drive1)//background.png; then 
          set color_normal=white/black 
          set color_highlight=black/white 
        else 
          set menu_color_normal=cyan/blue 
          set menu_color_highlight=white/blue 
        fi 

menuentry "NixOS - Default" --class nixos --unrestricted { 
search --set=drive1 --fs-uuid d6fa6ca3-71f6-4a65-8521-927494eded8c 
  linux ($drive1)//kernels/q2828ynf790bql5k3jgn0nham38p4gpl-linux-6.1.34-bzImage init=/nix/store/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634/init loglevel=4 
  initrd ($drive1)//kernels/5m6v3lymbzv0zz74iawlbdzs2r0hgzq6-initrd-linux-6.1.34-initrd ($drive1)//kernels/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634-secrets 
} 

submenu "NixOS - All configurations" --class submenu { 
menuentry "NixOS - Configuration 1 (2023-06-25 - 23.05.1249.ef0bc397634)" --class nixos { 
search --set=drive1 --fs-uuid d6fa6ca3-71f6-4a65-8521-927494eded8c 
  linux ($drive1)//kernels/q2828ynf790bql5k3jgn0nham38p4gpl-linux-6.1.34-bzImage init=/nix/store/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634/init loglevel=4 
  initrd ($drive1)//kernels/5m6v3lymbzv0zz74iawlbdzs2r0hgzq6-initrd-linux-6.1.34-initrd ($drive1)//kernels/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634-secrets 
} 

}

And here is the kexec line of heads IMG_20230625_210230_039

Can you please detail exactly how you did it so others can replicate and then we can find the issue and fix it?

I did it like this and I can replicate it:

And here is a grub config for an unencrypted Installation. Installation exactly as above except that encrypted isn't enabled on the root partition.

# Automatically generated.  DO NOT EDIT THIS FILE! 

    search --set=drive1 --fs-uuid 3d4b8a73-9032-4a2c-9dc7-abc75a28e405 
    if [ -s $prefix/grubenv ]; then 
      load_env 
    fi 

    # ‘grub-reboot’ sets a one-time saved entry, which we process here and 
    # then delete. 
    if [ "${next_entry}" ]; then 
      set default="${next_entry}" 
      set next_entry= 
      save_env next_entry 
      set timeout=1 
      set boot_once=true 
    else 
      set default=0 
      set timeout=5 
    fi 

    function savedefault { 
        if [ -z "${boot_once}"]; then 
        saved_entry="${chosen}" 
        save_env saved_entry 
        fi 
    } 

    # Setup the graphics stack for bios and efi systems 
    if [ "${grub_platform}" = "efi" ]; then 
      insmod efi_gop 
      insmod efi_uga 
    else 
      insmod vbe 
    fi 

        insmod font 
        if loadfont ($drive1)//converted-font.pf2; then 
          insmod gfxterm 
          if [ "${grub_platform}" = "efi" ]; then 
            set gfxmode=auto 
            set gfxpayload=keep 
          else 
            set gfxmode=1024x768 
            set gfxpayload=text 
          fi 
          terminal_output gfxterm 
        fi 

        background_color '#2F302F' 

        insmod png 
        if background_image --mode 'normal' ($drive1)//background.png; then 
          set color_normal=white/black 
          set color_highlight=black/white 
        else 
          set menu_color_normal=cyan/blue 
          set menu_color_highlight=white/blue 
        fi 

menuentry "NixOS - Default" --class nixos --unrestricted { 
search --set=drive1 --fs-uuid 3d4b8a73-9032-4a2c-9dc7-abc75a28e405 
  linux ($drive1)//kernels/q2828ynf790bql5k3jgn0nham38p4gpl-linux-6.1.34-bzImage init=/nix/store/xxchnmpjbarrk
r1inslh8vgzxk3byvzr-nixos-system-nixos-23.05.1249.ef0bc397634/init loglevel=4 
  initrd ($drive1)//kernels/yzrb4cyspnfgv2hxl616rw533m328ggc-initrd-linux-6.1.34-initrd 
} 

submenu "NixOS - All configurations" --class submenu { 
menuentry "NixOS - Configuration 1 (2023-06-27 - 23.05.1249.ef0bc397634)" --class nixos { 
search --set=drive1 --fs-uuid 3d4b8a73-9032-4a2c-9dc7-abc75a28e405 
  linux ($drive1)//kernels/q2828ynf790bql5k3jgn0nham38p4gpl-linux-6.1.34-bzImage init=/nix/store/xxchnmpjbarrk
r1inslh8vgzxk3byvzr-nixos-system-nixos-23.05.1249.ef0bc397634/init loglevel=4 
  initrd ($drive1)//kernels/yzrb4cyspnfgv2hxl616rw533m328ggc-initrd-linux-6.1.34-initrd 
} 

}
JonathonHall-Purism commented 1 year ago

Thanks @septem9er.

The encrypted setup has two initrds:

initrd ($drive1)//kernels/5m6v3lymbzv0zz74iawlbdzs2r0hgzq6-initrd-linux-6.1.34-initrd ($drive1)//kernels/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634-secrets

Heads understands this but then tries to pass two --initrd arguments to kexec, which doesn't work - kexec just takes the last one. This explains the error that's now visible, most of the initrd is missing, and since there's no init program Linux falls back to old logic to try to mount root on its own, which then fails, and the kernel panics with that error meaning it couldn't mount root.

kexec-boot will have to concatenate the two initrds, let me look into that.

tlaurion commented 1 year ago

@septem9er there is major security issue with NixOS installed on unencrypted /boot. The second initrd ($drive1)//kernels/bsfxa9xc6vgpr9nn6cl0mhnsxnmm3ssp-nixos-system-nixos-23.05.1249.ef0bc397634-secrets contains the LUKS deryption key to unlock the disk. You get it, that means that anyone having access to the machine can open that cpio and access your luks encrypted content.

Nix does vaguely warn you of something during the installation, but it's not very specific. It just says something about "important stuff is on /boot and won't be encrypted" and that "root will be decrypted later in the boot process" but does not tell you that it's storing a key to root in the clear on /boot

image

This is really wrong. If /boot is unencrypted, then the init scripts should ask for the decryption key passphrase on boot, not construct a cpio containing that secret lying forever on /boot unencrypted. Heads can and would, create such secret through setting a disk unlock key when defining a default disk option and selecting to setup a TPM unsealed disk unlock key, modifying/injecting a valid crypttab file and inject it through an additional cpio.

1426 would do the right thing here concatenating multiple cpios and injecting them as one at kexec call, but would still make that NixOS install insecure. NixOS should not install a secret on an unencrypted /boot.

Will tag NixOS devs pointing to this post.

JonathonHall-Purism commented 1 year ago

Nix developers are addressing this upstream, and it sounds like the intent is that the second initrd would not be present at all once this is fixed.

This renders the PR moot - though GRUB allows passing multiple initrds, I don't know of any other case where this happens and it is probably not worth the baggage to carry this as nearly-untestable code. Maybe Heads should just take the first one for simplicity, would have been less-wrong in this case.

@septem9er You might want to follow the discussion in the last Matrix link for updates, if Nix fixes this and the second initrd is gone, then this will work with PB/Heads as-is since the i915 graphics fix is already in.

@tlaurion If the fixes to Nix do eliminate the second initrd (not just leave it empty or something), then we can close #1426 and this issue IMO.

tlaurion commented 1 year ago

They are waiting for their security team, but here is one of the fixes proposed https://github.com/vlinkz/calamares-nixos-extensions/commit/837ca4da5521a74d3b5ca6f7b88890a6713faa22

RaitoBezarius commented 1 year ago

This was published as https://github.com/NixOS/calamares-nixos-extensions/security/advisories/GHSA-3rvf-24q2-24ww.

tlaurion commented 1 year ago

Testing iso here if you are faster then myself to test https://drive.google.com/file/d/1ZIUc2POI6Jdil56VITDlMib5qHVvpvpc/view?usp=drive_link

From https://matrix.to/#/!NBBFPbiuttRgTqbrcY:nixos.org/$w_XDBY3HOGcbfiEp0qnM9LbPbY8_VPW_px6f95EV5ek?via=matrix.org&via=nixos.dev&via=stratum0.org

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-36476

septem9er commented 1 year ago

Using this iso, I can boot into an encrypted NixOS instalation using heads. The initrd -secrets file isn't created.

Thanks for the help to both of you!

tlaurion commented 1 year ago

@septem9er will reopen until next official OS ISO is released?

@RaitoBezarius please close with comment when the next release of ISO with updated graphical installer happens?

tlaurion commented 1 year ago

@RaitoBezarius can we close?

RaitoBezarius commented 1 year ago

Sorry for this, @vlinkz can you confirm this? I believe this is the case, but just in case.

vlinkz commented 1 year ago

Sorry for this, @vlinkz can you confirm this? I believe this is the case, but just in case.

We need to merge this first: https://github.com/NixOS/nixpkgs/pull/247163

tlaurion commented 1 year ago

@septem9er @vlinkz @RaitoBezarius

Sorry for this, @vlinkz can you confirm this? I believe this is the case, but just in case.

We need to merge this first: NixOS/nixpkgs#247163

This is merged. Can we close?

RaitoBezarius commented 1 year ago

Correct!

Nintorac commented 3 months ago

This may have regressed in the 24.05 graphical installer (specifically nixos-gnome-24.05.2411.706eef542dec-x86_64-linux.iso)

I have a similar setup with an unencrypted /boot and an encrypted /. When I start the OS it just boots all the way to user login not requiring my decryption password at any time.

Here is my /boot/kernels, including the secrets one that tlaurion mentioned being problematic. image

I don't have the knowledge to validate my secret is in here though

septem9er commented 3 months ago

Hey @Nintorac, thank you for the report. This is a NixOS issue, so here is not really the right place to report it. However, I was able to reproduce this issue and just now reported it to the NixOS security Team via Matrix. https://matrix.to/#/!ZRgXNaHrdpGqwUnGnj:nixos.org/$jONbiFhqF-8kEvzdhIkMN592cSWn0PPrSKEt4PhCESw?via=nixos.org&via=matrix.org&via=nixos.dev

Nintorac commented 3 months ago

ooh my bad, didn't even look where I was :!

Awesome, thanks!!

tlaurion commented 3 months ago

Ongoing discussion around https://matrix.to/#/!NBBFPbiuttRgTqbrcY:nixos.org/$OP4Zb70brQuqRGpiPLpVhq2-iocmRydo8lkbSBEFKbg?via=matrix.org&via=tchncs.de&via=nixos.dev

tlaurion commented 3 months ago

Switched to https://matrix.to/#/!NBBFPbiuttRgTqbrcY:nixos.org/$6hZzQzfqcjGYxj1qtlGDkKjrBZbf0sikdYCnbGNqx-U?via=matrix.org&via=tchncs.de&via=nixos.dev

emilazy commented 2 months ago

We’ve fixed this and posted an advisory at https://github.com/NixOS/calamares-nixos-extensions/security/advisories/GHSA-vfxf-gpmj-2p25. Sorry that we let this regression slip through the cracks and that it took a while to address it; we have less in‐house Calamares expertise than I’d like. Thanks to @dalto8 for helping us review the fix.