linux 5.15.99 freezes at i915 module loading, breaks boot

[schmittlauch]

Issue description

After updating my NixOS 22.11 system regularly, I discovered that it does not boot anymore:

The NixOS stage 1 gets stuck after loading module i915.... The system does not react anymore to output other than pressing the main power button, e.g. a Ctrl+Alt+Del is ignored as well.

At a successful boot, as shown below, the next module to be loaded is Btrfs, as this is the filesystem for both my / (my /boot/ partition is using vfat though):

By bisecting the nixpkgs release-22.11 branch, I was able to bisect this issue to the linux kernel update from 5.15.97 -> 5.15.99:

git bisect start
# Status: warte auf guten und schlechten Commit
# bad: [b628dd9754bb9ba46e5bb4a4775fb00db12e68ce] Merge pull request #220640 from LeSuisse/nifi-22.11-vulnerabilities
git bisect bad b628dd9754bb9ba46e5bb4a4775fb00db12e68ce
# Status: warte auf gute(n) Commit(s), schlechter Commit bekannt
# good: [96e18717904dfedcd884541e5a92bf9ff632cf39] Merge pull request #217694 from NixOS/backport-214746-to-release-22.11
git bisect good 96e18717904dfedcd884541e5a92bf9ff632cf39
# good: [91356c6fb2c02e1dd9db82e276b1173079500399] [22.11] python3Packages.werkzeug: 2.2.2 -> 2.2.3 (#220050)
git bisect good 91356c6fb2c02e1dd9db82e276b1173079500399
# bad: [5eb98948b66de29f899c7fe27ae112a47964baf8] Merge pull request #218966 from NixOS/backport-218860-to-release-22.11
git bisect bad 5eb98948b66de29f899c7fe27ae112a47964baf8
# good: [db48f250c80810d639de9df073670f431784bb93] Merge pull request #220432 from NixOS/backport-220386-to-release-22.11
git bisect good db48f250c80810d639de9df073670f431784bb93
# bad: [65c58da9fb170d2789806ee3847ef0d26b6433f4] linux/hardened/patches/5.15: 5.15.96-hardened1 -> 5.15.98-hardened1
git bisect bad 65c58da9fb170d2789806ee3847ef0d26b6433f4
# good: [824f886682fc893e6dbf27114e5001ebf2770ea1] Merge pull request #219967 from NixOS/backport-219803-to-release-22.11
git bisect good 824f886682fc893e6dbf27114e5001ebf2770ea1
# bad: [db6b668d1d778c698873fe4c27a9c971a6ceeabc] linux: 6.2 -> 6.2.3
git bisect bad db6b668d1d778c698873fe4c27a9c971a6ceeabc
# bad: [4e229f4beceb7e8a5ae81f9b96ff7955fa45d3be] linux: 6.1.15 -> 6.1.16
git bisect bad 4e229f4beceb7e8a5ae81f9b96ff7955fa45d3be
# bad: [c022e1da3c960fc68fc4145fdcf10fd502de9d41] linux: 5.15.97 -> 5.15.99
git bisect bad c022e1da3c960fc68fc4145fdcf10fd502de9d41
# first bad commit: [c022e1da3c960fc68fc4145fdcf10fd502de9d41] linux: 5.15.97 -> 5.15.99

Steps to reproduce

You need a NixOS installation booting directly from a Btrfs partition, like done in my system configuration. The rest of my system config can be found here

Next steps: I'll try to gather more detailed kernel logs within the next days and report back, still feel free to ask for specific log information. Additionally, kernel 5.15.101 has already been released upstream (but not yet reached release-22.11). I'll try whether the issue is fixed there. I'll try the same with the latest 6.2.x kernel release.

Technical details

hardware: Lenovo Thinkpad T440S CPU: i7-4600U -> graphics: Intel HD 4400

Note: The output below resembles the state of my system when built on and booted from the last commit where the issue did not appear, as I of course could not run it from a non-booting system.

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.15.97, NixOS, 22.11 (Raccoon), 22.11.3042.5eb98948b66
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.11.1
• channels(root): "nixos-22.11, nixos-unstable"
• channels(user): "NUR, home-manager-22.11.tar.gz, nixos-22.11"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[paulwalker-dev]

Does the "nomodeset" kernel parameter allow you to get past this screen?

[bbjubjub2494]

Same issue here. Also using BTRFS and ZFS. Does appear to be due to the kernel update. I don't know yet if my desktop is affected.

[paulwalker-dev]

This issue is also happening on a system with ext4 as root.

[schmittlauch]

@paulwalker-dev I can confirm that giving the nomodeset cmdline indeed fixes the boot. So this appears to be more related to the i915 module not loading correctly and freezing, thus blocking the loading of later modules.

My affected graphics chip: Intel HD 4400 on a core i7-4600U

Kernel 6.2.3 is booting just fine. I'll try earlier and later releases of the 5.15.x branch next.

[woffs]

maybe related to https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.15.101

[bbjubjub2494]

i7-2600 affected. linux 5.15.97 not affected.

Putting i915 in boot.blacklistedModules allows booting anyways with no iGPU functionality, so it is i915.

Now trying newer kernel versions.

[paulwalker-dev]

220909 Linux 5.15.101 is just waiting to be backported to the nixos-22.11 branch

[schmittlauch]

Good to know, I am still busy building 5.15.102 as this is the latest release from that branch.

[schmittlauch]

I can confirm that Linux 5.15.102 also still fixes the issue. Given that 5.15.101, which supposedly also fixes this, is already in the nixos-22.11-small channel, I guess we can close this.

[erictapen]

So this was a chipset specfic issue? Probably hard to automatically test against this I assume?

[manassingh2611]

A temporary fix can be adding "nomodeset i915.modeset=1" in grub params

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/override-attribute-in-nixos-hardware-expression/38426/1