search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.66k stars 13.81k forks source link

NixOS on Raspberry Pi 4 Model B | bluetoothctl issue: No default controller available #123725

Open yshym opened 3 years ago

yshym commented 3 years ago

Issue description

"No default controller available" issue while trying to use bluetoothctl tool. I've recently installed NixOS on a Raspberry Pi 4 Model B using this very cool tutorial and everything seems to be fine for now except for this bluetooth functionality. I've read that https://github.com/RPi-Distro/pi-bluetooth is used in most similar cases, but, as far as I understand, btattach systemd service (https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_3#Bluetooth) is already doing the same (if it is suitable for both rpi 3 and 4)

rpi-related imports:

imports = [
  "${fetchTarball "https://github.com/NixOS/nixos-hardware/archive/936e4649098d6a5e0762058cb7687be1b2d90550.tar.gz" }/raspberry-pi/4"
];

systemd services:

systemd.services = {
  btattach = {
    before = [ "bluetooth.service" ];
    after = [ "dev-ttyAMA0.device" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      ExecStart = "${pkgs.bluez}/bin/btattach -B /dev/ttyAMA0 -P bcm -S 3000000";
    };
  };
};

hardware:

hardware = {
  # Enable GPU acceleration
  raspberry-pi."4".fkms-3d.enable = true;
  enableRedistributableFirmware = true;
  pulseaudio = {
    enable = true;
    support32Bit = true;
    extraModules = [ pkgs.pulseaudio-modules-bt ];
    package = pkgs.pulseaudioFull;
    extraConfig = ''
      load-module module-switch-on-connect
    '';
  };
  bluetooth.enable = true;
};
rfkill list all
0: hci0: Bluetooth
    Soft blocked: no
    Hard blocked: no
1: phy0: Wireless LAN
    Soft blocked: no
    Hard blocked: no

Steps to reproduce

bluetoothctl
Agent registered
[bluetooth]# power on
No default controller available

Technical details

domenkozar commented 3 years ago

Looking at google results it seems that some kernels broke Bluetooth support, so updating kernel to the latest revision in style of https://github.com/NixOS/nixpkgs/pull/118548 might fix it.

yshym commented 3 years ago

fkms-3d overlay can't be built with boot.kernelPackages = pkgs.linuxPackages_latest;:

Applying overlay rpi4-vc4-fkms-v3d-overlay to bcm2711-rpi-4-b.dtb

Failed to apply '/nix/store/igz72xqn4w32hgbgiz8ab4493kj87j63-rpi4-vc4-fkms-v3d-overlay-dtbo': FDT_ERR_NOTFOUND
builder for '/nix/store/5ma90bwh3j5ljg5p078yf8x5h19m7ixn-device-tree-overlays.drv' failed with exit code 1
cannot build derivation '/nix/store/amm3mv4c7hy203gbx9w1y3i044phry7w-nixos-system-rpi4-21.05pre-git.drv': 1 dependencies couldn't be built
domenkozar commented 3 years ago

Sorry, I meant packaging the latest raspberrypi kernel revision.

yshym commented 3 years ago

Oh, ok. I'll try it

yshym commented 3 years ago

A bit newer release doesn't fix the problem. Latest revisions from both rpi-5.11.y and rpi-5.12.y have compilation issues

nixos-discourse commented 3 years ago

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/raspberry-pi-4-bluetooth-audio/13876/2

colonelpanic8 commented 3 years ago

A bit newer release doesn't fix the problem. Latest revisions from both rpi-5.11.y and rpi-5.12.y have compilation issues

What are the compilation issues? Are they not fixable?

VergeDX commented 2 years ago

Same issues here, I tried https://github.com/raspberrypi/linux/releases/tag/1.20211007 version.

bbqsrc commented 2 years ago

This remains to be an issue. Have hit this today using the pkgs.linuxPackages_rpi4 kernel.

pschyska commented 2 years ago

I noticed that console=ttyAMA0,115200n8 in my kernel commandline (via sd-image.nix) breaks bluetooth. I had a number of dmesg lines about hc0, e.g. hci0 command 0x0c03 tx timeout. Removing the ttyAMA0 entry made bluetooth work with something like

{
  imports = [  (modulesPath + "/installer/sd-card/sd-image-aarch64.nix") ];
  hardware = {
    bluetooth = {
      package = pkgs.bluez;
      enable = true;
      powerOnBoot = false;
    };
  };
}

nixpkgs@ 35db2e60df598e3d5c88adab65f70b91f6645b45, linuxPackages_latest.kernel.version = 5.16.12

I edited the kernel command line in extlinux.conf by hand for now, nixified it would probably look like this:

{
  # via sd-image: boot.kernelParams = ["console=ttyS0,115200n8" "console=ttyAMA0,115200n8" "console=tty0"];
  boot.kernelParams = lib.mkForce ["console=ttyS0,115200n8" "console=tty0"];
}
Majiir commented 2 years ago

@pschyska Could you share a full config with working bluetooth? And just in case, could you also share your /boot/config.txt? I have been unable to get bluetooth working by downgrading kernels or adopting any of the configs in this thread. If I manually run sudo btattach -B /dev/ttyAMA0 -P bcm -S 3000000, I can see the adapter with rfkill:

ID TYPE      DEVICE      SOFT      HARD                                                                                                                        │
 1 bluetooth hci0   unblocked unblocked

but bluetoothctl power on still fails with No default controller available. I do see these errors in dmesg upon running btattach:

[  525.931400] Bluetooth: hci0: command 0x0c03 tx timeout                                                                                                      │
[  533.995470] Bluetooth: hci0: BCM: Reset failed (-110)

but I do not have the problematic kernel command line params you referenced.

pschyska commented 2 years ago

@Majiir I'm afraid I can't. I switched to https://github.com/pftf/RPi4 and install nixos like on any other uefi system. I couldn't make the bluetooth controller appear in bluetoothctl any longer (the resulting installation is probably all kinds of broken now in other ways, but I have no clue about rpis). As I didn't end up using bluetooth I didn't notice. With the snippet above I did get it to work and was able to pair a device and didn't see any issues. I also can't experiment much because I need my rpi working for some network functions. Sorry

pschyska commented 2 years ago

@Majiir I recently worked on my system again, and was able to use bluetooth with upstream kernels 5.15.63 (linuxPackages @ nixos-unstable) and 5.19.4 (linuxPackages_latest). I didn't do anything special. I'm not importing nixos-hardware.

config.txt:

arm_64bit=1
enable_uart=1
uart_2ndstage=1
enable_gic=1
armstub=RPI_EFI.fd
disable_commandline_tags=1
disable_overscan=1
device_tree_address=0x1f0000
device_tree_end=0x200000
dtoverlay=miniuart-bt
#dtoverlay=upstream-pi4

(upstream-pi4 always crashed the system during boot) EEPROM is https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/stable/pieeprom-2022-08-02.bin RPI_EFI and the boot code is from https://github.com/pftf/RPi4/releases/tag/v1.33. I have a 4 B rev 1.1 with 4G RAM and removed the 3G limit in UEFI and set the boot mode to ACPI+DeviceTree (with just ACPI I'm missing a lot of functionality like raspberrypi_cpufreq).

relevant nix config:

{pkgs, ...}: {
  boot = {
    loader = {
      timeout = 3;

      # requires uefi firmware!
      systemd-boot = {
        enable = true;
        consoleMode = "max";
      };
      efi.canTouchEfiVariables = true;
      generic-extlinux-compatible.enable = false;
    };
    kernelPackages = pkgs.linuxPackages_latest; # or without _latest
    # kernelParams not set, defaults to = ["loglevel=4"];

    initrd.availableKernelModules = [ "pcie-brcmstb" "reset-raspberrypi" ]; # USB hid in initrd to unlock luks devices
  };

  hardware.enableRedistributableFirmware = true;
  hardware.deviceTree.filter = "bcm2711-rpi-4-b.dtb";
  systemd.services."getty@".enable = false;
  powerManagement.cpuFreqGovernor = "schedutil";
}

Bear in mind that depending on you UART settings, the bluetooth device is either /dev/ttyAMA0, or one of the /dev/ttyS (IIRC it was /dev/ttyS1 for me with miniuart-bt, but for most it's /dev/ttyS0, IDK why it moved to the 1 for me ¯\(ツ)/¯). In either case, make sure you don't put a linux console onto this device (e.g. with the referenced kernel params, or via enabling getty@tty or similar). If you use mini UART, you have to update btattach -B /dev/ttyS0 -P bcm -S 3000000 accordingly. Afterwards I systemctl restart bluetooth and could the the controller in bluetoothctl, and could power on and scan on and got results.

I can't give you the full config, is split up into many modules within my system flake, but I can query specific config values in repl for you if you want to know more.

Majiir commented 2 years ago

Thanks, @pschyska. I tried pieces of your config, and I narrowed the issue down to, believe it or not, the kernel: linuxPackages works fine, but linuxPackages_rpi does not.

(Also, I'm using sd-image-aarch64.nix as a base, so I had to set kernelParams = lib.mkForce [ "console=ttyS0,115200n8" "console=tty0" ] in order to remove the default console=ttyAMA0,115200n8 parameter.)

I don't think the issue is in the kernel itself, since I had bluetooth working when running the same kernel version on Raspberry Pi OS. I'll run some more experiments to verify one way or the other.

pschyska commented 2 years ago

Thanks, @pschyska. I tried pieces of your config, and I narrowed the issue down to, believe it or not, the kernel: linuxPackages works fine, but linuxPackages_rpi does not.

I assume you mean linuxPackages_rpi4 and it's not hard to believe that this is the issue 🙁 Raspberry foundation are doing some things that upstream kernel doesn't agree with, IIUC, beginning with the naming of the SoC in DeviceTree (e.g. for pi3-b, rpi foundation calls it bcm2710-rpi-3-b. and upstream kernel bcm2837-rpi-3-b, although it looked like at least for the 4's both name it bcm2711, but I'm no expert...) and probably custom kernel modules, firmware blobs and so on. Did you take a look at nixos-hardware/raspberry-pi/4 and its includes? This config is supposed to work with linuxPackages_rpi4. In particular, you need hardware.enableRedistributableFirmware so that you get pkgs.raspberrypiWirelessFirmware.

ChanceHarrison commented 1 year ago

Is btattach a requirement to get the Pi 4B's Bluetooth working (once everything is configured properly)? If so, would it make sense to "fix" this in nixos-hardawre with some combination of udev rules and systemd services to start btattach automatically?

Majiir commented 1 year ago

There's also the krnbt device tree parameter:

        krnbt                   Set to "on" to enable autoprobing of Bluetooth
                                driver without need of hciattach/btattach
                                (default "off")

I read that krnbt is the preferred approach, but I can't recall where I read that. It's certainly simpler, when it works.

Thing is, I haven't been able to get btattach to work either. To summarize, these are the results I'm seeing:

ChanceHarrison commented 1 year ago

@Majiir Thanks for the mention of krnbt; I didn't know it existed.

I have btattach and NixOS with the Raspberry Pi kernel working. Funny enough, I can't boot with the default kernel.

Here is what works for me.

configuration.nix

boot.kernelParams = [
  "8250.nr_uarts=1"
  "console=ttyS0,115200"
  "console=tty1"
];
boot.kernelPackages = pkgs.linuxPackages_rpi4;

hardware.bluetooth.enable = true;

(I think those are the only relevant lines)

config.txt

[pi4]
kernel=Tow-Boot.noenv.rpi4.bin
enable_gic=1
armstub=armstub8-gic.bin
disable_overscan=1

[all]
arm_64bit=1
enable_uart=1
avoid_warnings=1

nix-info -m

Perhaps that can help you narrow down what isn't working for you? Happy to provide any extra information.

Majiir commented 1 year ago

@ChanceHarrison Could you share the btattach command you are using?

ChanceHarrison commented 1 year ago

Of course! I knew I was forgetting something.

btattach -B /dev/ttyAMA0 -P bcm

ChanceHarrison commented 1 year ago

There's also the krnbt device tree parameter: - @Majiir

Edit: It works just fine for me, if you type it correctly. 😉

Original text: For what it's worth, adding dtparm=krnbt=on to my config.txt and rebooting didn't change anything; I still had to use btattach to get bluetoothctl to see the controller. Am I doing something obviously wrong? Do I also have to set the krnbt_baudrate param? I'm not well-versed in debugging device tree shenanigans.

Majiir commented 1 year ago

I'm not well-versed in it either, but I only needed the dtparam=krnbt=on line. That worked for me in NixOS (with the default kernel) and also in Raspberry Pi OS (with the btattach systemd units disabled).

Your post says dtparm, but it should be dtparam - did the same typo make it into your config?

ChanceHarrison commented 1 year ago

Your post says dtparm, but it should be dtparam - did the same typo make it into your config?

D'oh! The typo made its way into the config first, actually. Thank you for catching it. It works as advertised if you actually type it correctly.

I think there are some lessons learned here that can be documented on the wiki, perhaps? I would be happy to try my hand at those changes, though I have never done so before.

As for your issue, did you have any luck finding why Bluetooth (either via btattach or krnbt) is working for me and not for you?

Majiir commented 1 year ago

No such luck! I tried using the same nixpkgs commit, the same NixOS config, and the same config.txt contents. The behavior is the same every time: btattach runs, and rfkill shows a device:

$ rfkill
ID TYPE      DEVICE      SOFT      HARD
 0 bluetooth hci0   unblocked unblocked

but still no working Bluetooth:

$ bluetoothctl power on
No default controller available

dmesg reports:

[  204.090325] uart-pl011 fe201000.serial: no DMA platform data
[  204.116448] Bluetooth: HCI UART driver ver 2.3
[  204.120994] Bluetooth: HCI UART protocol H4 registered
[  204.126231] Bluetooth: HCI UART protocol BCSP registered
[  204.131728] Bluetooth: HCI UART protocol LL registered
[  204.137010] Bluetooth: HCI UART protocol Three-wire (H5) registered
[  204.143629] Bluetooth: HCI UART protocol Broadcom registered
[  204.149465] Bluetooth: HCI UART protocol QCA registered
[  204.317159] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[  204.322579] Bluetooth: BNEP filters: protocol multicast
[  204.328044] Bluetooth: BNEP socket layer initialized
[  206.174626] Bluetooth: hci0: command 0x0c03 tx timeout
[  214.238705] Bluetooth: hci0: BCM: Reset failed (-110)

In all of my testing, the only thing that works is switching to the default kernel instead of linuxPackages_rpi4. But the issue isn't the Raspberry Pi kernel itself, because that kernel works with both manual attachment and krnbt on other distros.

I put together a minimal-ish system flake for testing this: https://github.com/Majiir/btpi You can build an SD image with nix build --flake '.#btpi-sd, or you can rebuild an existing system with nixos-rebuild --flake '.#btpi'.

@ChanceHarrison Would you be willing to build a fresh SD image and give that a try? If it works, then that narrows the issue down to hardware or firmware. If it doesn't, then maybe you can make adjustments or offer an alternative flake that does work.

[EDIT] Just to rule out this one device being the problem, I tested the btattach method on a Raspberry Pi Compute Module 4 with a similar config, and I'm seeing the same issue.

ChanceHarrison commented 1 year ago

Would you be willing to build a fresh SD image and give that a try? - @Majiir

I would be willing to give it a try, yes. I'll get back to you in a few days once I have a new spare SD card to use. To make sure I'm clear on the process:

  1. Grab https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux, dd onto SD, and boot
  2. git clone https://github.com/Majiir/btpi and cd btpi
  3. nixos-rebuild --flake '.#btpi'

Or, would I just do step 2 (on an existing NixOS system) and nix-build --flake '.#btpi-sd' and then dd output (where does the output go?) onto SD and boot? Excuse my lack of understanding there, but does /etc/nixos/configuration.nix get populated when building an SD image this way?

Of note, I am not using the SD image to get NixOS on my Pi. I used the aarch64 minimal installer (specific version nixos-minimal-22.11pre409157.da6a05816e7-aarch64-linux.iso, though I don't know how to find the build on Hydra) that I wrote to a USB flash drive. I wrote Tow-Boot (the shared.disk-image.img from raspberryPi-aarch64-2021.10-005) on the SD card for UEFI booting. From there, I installed NixOS onto a SSD (also connected via USB). Do you think it's possible that the differences in our observed behavior are because I'm not using the SD image and you are? I suppose we might find out when I give the SD image route a try myself.

Majiir commented 1 year ago

We have a few different options. For best reproducing my setup, you should do something like this:

git clone https://github.com/Majiir/btpi
cd btpi
nix build '.#btpi-sd'
ls result/sd-image

Then, you can write that image to an SD card, boot the Pi, and log in with test/test.

The SD image will create a config.txt that contains dtparam=krnbt=on, but that can be manually removed by mounting the firmware partition (mount /dev/disk/by-label/FIRMWARE) and editing /boot/firmware/config.txt. It shouldn't make a difference to the test (except that, y'know, Bluetooth might just work when you boot up).

If you do all that, you won't have anything in /etc/nixos. You can clone the repo to the Pi and nixos-rebuild switch --flake '.#btpi' there, or you can remotely deploy from another machine with --target-host.

My thinking is that by using a flake and building the whole SD image, we ideally get the exact same setup, and Bluetooth should fail for you with that setup. If it does, you could try taking your existing, working system (after taking a backup!) and deploy the minimal flake with nixos-rebuild. The results will be interesting! Thank you for taking the time to test this out.

Meanwhile, I will see if I can replicate your install steps.

ChanceHarrison commented 1 year ago

@Majiir I finally got my spare SD card(s) and I went ahead and booted my Pi with your minimal example. Thanks for the guidance on using it. Sure enough, I run into the same Bluetooth issues. Adding hardware.enableRedistributableFirmware = true; and rebuilding the system with the flake didn't help.

I compared dmesg output between my known-good system and the minimal example system. Here's the two outputs:

Known good:

[   11.921327] Bluetooth: HCI device and connection manager initialized
[   11.921345] Bluetooth: HCI socket layer initialized
[   11.921364] Bluetooth: L2CAP socket layer initialized
[   11.921383] Bluetooth: SCO socket layer initialized
[   12.073632] Bluetooth: HCI UART driver ver 2.3
[   12.073653] Bluetooth: HCI UART protocol H4 registered
[   12.073658] Bluetooth: HCI UART protocol BCSP registered
[   12.073741] Bluetooth: HCI UART protocol LL registered
[   12.076474] Bluetooth: HCI UART protocol Three-wire (H5) registered
[   12.079804] Bluetooth: HCI UART protocol Broadcom registered
[   12.441954] Bluetooth: hci0: BCM: chip id 107
[   12.442231] Bluetooth: hci0: BCM: features 0x2f
[   12.443316] Bluetooth: hci0: BCM4345C0
[   12.443335] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[   12.456244] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[   13.144837] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   13.144857] Bluetooth: BNEP filters: protocol multicast
[   13.144877] Bluetooth: BNEP socket layer initialized
[   13.167354] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+
[   13.167377] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0342

btpi (with enableRedistributableFirmware):

[    9.122898] Bluetooth: Core ver 2.22
[    9.140820] Bluetooth: HCI device and connection manager initialized
[    9.141508] Bluetooth: HCI socket layer initialized
[    9.141536] Bluetooth: L2CAP socket layer initialized
[    9.141972] Bluetooth: SCO socket layer initialized
# btattach
[  437.675481] uart-pl011 fe201000.serial: no DMA platform data
[  437.705865] Bluetooth: HCI UART driver ver 2.3
[  437.710439] Bluetooth: HCI UART protocol H4 registered
[  437.715687] Bluetooth: HCI UART protocol BCSP registered
[  437.721161] Bluetooth: HCI UART protocol LL registered
[  437.726454] Bluetooth: HCI UART protocol Three-wire (H5) registered
[  437.733011] Bluetooth: HCI UART protocol Broadcom registered
[  437.738863] Bluetooth: HCI UART protocol QCA registered
[  439.777322] Bluetooth: hci0: command 0x0c03 tx timeout
[  447.969540] Bluetooth: hci0: BCM: Reset failed (-110)

I also confirmed that switching to the non-rpi4 kernel (I tried with linuxPackages_latest) works just fine:

[   14.071134] Bluetooth: Core ver 2.22
[   14.071328] Bluetooth: HCI device and connection manager initialized
[   14.071352] Bluetooth: HCI socket layer initialized
[   14.071363] Bluetooth: L2CAP socket layer initialized
[   14.071404] Bluetooth: SCO socket layer initialized
[   14.397969] Bluetooth: HCI UART driver ver 2.3
[   14.402607] Bluetooth: HCI UART protocol H4 registered
[   14.407929] Bluetooth: HCI UART protocol BCSP registered
[   14.490109] Bluetooth: HCI UART protocol LL registered
[   14.497315] Bluetooth: HCI UART protocol Broadcom registered
[   14.565484] Bluetooth: HCI UART protocol QCA registered
[   14.573000] Bluetooth: HCI UART protocol Marvell registered
[   14.912586] Bluetooth: hci0: BCM: chip id 107
[   14.922423] Bluetooth: hci0: BCM: features 0x2f
[   14.928848] Bluetooth: hci0: BCM4345C0
[   14.928870] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[   14.946924] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[   15.230121] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   15.235649] Bluetooth: BNEP socket layer initialized
[   15.765508] Bluetooth: hci0: BCM: features 0x2f
[   15.771862] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+
[   15.777712] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0342
[   15.802863] Bluetooth: MGMT ver 1.22

So now I am very interested how I have my known-good system working with the rpi4 kernel but I can't get it to boot with the normal kernel...

I haven't tried

taking your existing, working system (after taking a backup!) and deploy the minimal flake with nixos-rebuild

Is relying on being able to boot the previous generation a sufficient backup? 🤔

I hope we can work together to figure this out. Hopefully others will chime in if they know something or have any suggestions.

FPtje commented 1 year ago

I'm a newcomer to this issue. Today I was playing with my newly acquired Raspberry Pi 4B, 4GB and played around with this issue. I would like to confirm this bug with the current nixpkgs, and I would like to provide a summary of the current status:

My nixpkg revision is 5dc7114b7b256d217fe7752f1614be2514e61bb8. It's nixpkgs-unstable, pin updated at 27-11-2022.

I played with a couple of kernels. A summary is below:

Raspberry Pi Kernel

Following the wiki and the linked nix.dev article sets up an install using the generic aarch64 SD image. After following these guides, one ends up importing nixos-hardware.nixosModules.raspberry-pi-4 from the NixOS-hardware repo. This setup will use the Linux kernel as provided by the Raspberry foundation (attribute name pkgs.linuxPackages_rpi4).

This setup has the following problems:

Upstream kernel 5.15

One can use kernel 5.15, by setting boot.kernelPackages = pkgs.linuxPackages; (or pkgs.linuxPackages_5_15 to be precise) in the NixOS config (as of 5dc7114b7b256d217fe7752f1614be2514e61bb8). This changes the following:

Here is a (reduced) version of the config that works for me:

{ config, pkgs, lib, ... }:
{
  hardware = {
    bluetooth.enable = true;
    enableRedistributableFirmware = true;
    pulseaudio.enable = true;
    deviceTree.filter = "bcm2711-rpi-*.dtb";
  };

  boot = {
    # Bluetooth doesn't work with rpi4 kernel
    # https://github.com/NixOS/nixpkgs/issues/123725
    kernelPackages = pkgs.linuxPackages;
    # Not sure how much of this this is relevant
    initrd.availableKernelModules = [
      "usbhid"
      "usb_storage"
      "vc4"
      "pcie_brcmstb"      # required for the pcie bus to work
      "reset-raspberrypi" # required for vl805 firmware to load
    ];
    loader = {
      generic-extlinux-compatible.enable = true;
    };
  };

  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
      options = [ "noatime" ];
    };
  };
}

Some elements taken from nixos-hardware. I copied those over manually because I load that repo in through a flake, which doesn't make for a nice snippet.

Upstream kernel 6.0.9

This kernel can be used by setting boot.kernelPackages = pkgs.linuxPackages_latest; (or pkgs.linuxPackages_6_0 to be precise). This kernel failed to boot to desktop, as the x-server started to SEGFAULT on boot.

Edit 28-11: interestingly, this article states that video driver support was merged for kernel version 6, but implies there are some bugs. That may explain the SEGFAULTs I'm seeing. I haven't looked further into this.

Kernels between 5.15 and 6.0.9 have dropped upstream support, and have been removed from nixpkgs. Therefore I haven't tried them.

Edit: I've added a link to this issue in the Wiki page: https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4

avieth commented 1 year ago

One way to get bluetooth to work is by copying the firmware and kernel from a raspbian image and then hacking it into NixOS. Here's how.

lilyball commented 1 year ago

@avieth If I already have a configured raspberry pi, is there a way to update it in place with this?

avieth commented 1 year ago

@avieth If I already have a configured raspberry pi, is there a way to update it in place with this?

The typical update step from the README on the page I linked should work fine since it's just a typical switch-to-configuration

$ nix-build rpi4.nix --argstr hostName <your hostname> -A update -o update
$ nix-copy-closure --to <pi address> ./update
$ ssh <pi address> $(readlink update)

What's missing is the bootloader setup and it's a bit weird. You could build the initial image just like in the README

$ nix-build rpi4.nix --argstr hostName <your hostname> -A initial -o initial

and then mount the boot partition of the image

sudo mount -o loop,offset=$((16384*512)) ./sd-image/sd-image/nixos-sd-image-21.11pre-git-aarch64-linux.img <mount point>

and replace your existing Pi's boot with the contents of this. But also edit cmdline.txt to change the root and rootfstype parameters accordingly. It's set to use PARTUUID=ffffffff-2 and the sd-image is set to match but that's probably not going to match your setup.

tstat commented 1 year ago

The firmware does some manipulation before passing the device tree to the kernel. The device tree in the raspberry pi kernel seems to expect this manipulation to happen, as the device tree in their fork isn't fully configured like the one in the mainline kernel. This is problematic since NixOS does not (by default) use the device tree provided by the firmware and instead supplies its own.

In this case I believe the problem is that the firmware configures the bluetooth flow control pins. I have fixed the bluetooth issue in the past by creating an overlay that mimics the firmware manipulation. Something like:

// Configure uart0 pins so that bluetooth works
/dts-v1/;
/plugin/;

/ {
    compatible = "brcm,bcm2835";

    fragment@0 {
        target = <&uart0_pins>;
        __overlay__ {
                brcm,pins = <30 31 32 33>;
                brcm,pull = <2 0 0 2>;
        };
    };
};

It is tedious and error prone to maintain these patches. So, more recently I have pursued using the device tree that the firmware provides (using u-boot's CONFIG_OF_BOARD option) rather than trying to mimic the manipulation. This approach has worked well for me.

I packaged up this approach here and an example with working bluetooth can be found here.

lilyball commented 1 year ago

@tstat That looks promising. Is there any guide to migrating an existing deploy onto your raspberry-pi-nix approach? I don't know much at all about the Linux booting process and I'm leery about just changing the way an existing system boots because if it gets screwed up I don't know how to fix it.

Right now my system is configured to use nixos-hardware.nixosModules.raspberry-pi-4 and then sets boot.kernelPackages = pkgs.linuxPackages. It looks like this is using boot.loader.generic-extlinux-compatible (that's what nixos-hardware sets, though I've noticed there's a separate boot.loader.raspberryPi and I don't know what the difference is or why nixos-hardware doesn't set that).

tstat commented 1 year ago

There isn't a guide for migrating an existing deploy. In your case it sounds the steps would be:

  1. Add the raspberry-pi-nix module from the v0.2.0 release
  2. Remove the nixos-hardware module
  3. Remove boot.kernelPackages = pkgs.linuxPackages since raspberry-pi-nix boots a recent raspberry-pi kernel fork
  4. If you have customized your config.txt file then port those customizations to your nixos configuration

If you are using nixos-hardware now then migrating shouldn't change the way your system boots; nixos-hardware also boots with u-boot on the firmware partition, which then locates /boot/extlinux/extlinux.conf on your root partition to discover the kernel location and whatnot.

If you want to be cautious you could always back up an image of your sd-card before attempting, or you could be less cautious and only backup your firmware partition as that is destructively modified, whereas your root partition is not (edit: I just recalled that modifying boot.loader.generic-extlinux-compatible.useGenerationDeviceTree is destructive when changed, so being cautious you might want to back up /boot/extlinux/extlinux.conf as well). Feel free to open an issue on raspberry-pi-nix if you have other questions or want to discuss creation of a migration guide.

The readme of raspberry-pi-nix attempts to describe how it works and motivate the design, which might be helpful or interesting to you.

If you end up attempting a migration please let me know how it goes!

lilyball commented 1 year ago

@tstat I just attempted it tonight, but I ran into a roadblock which is that compiling the kernel requires the big-parallel feature and I don't have that (I'm using a hacky Docker setup to build for aarch64-linux so I don't have to build on my rpi). Is there any binary cache available?

tstat commented 1 year ago

raspberry-pi-nix is comprised solely of nixos modules and does not pin a version of nixpkgs (i.e. nixpkgs is not an input to the raspberry-pi-nix flake), so I can't guarantee any cache hits. However, I built the kernel and other packages defined in the overlay against nixos-22.11 and pushed that to cachix. The example in the readme now shows the appropriate substituter and trusted key settings.

lilyball commented 1 year ago

@tstat Thanks for trying! Unfortunately it's not working for me. I'm guessing this is because I'm doing this on aarch64-darwin and based on the narHash issue I filed on your repo it looks like the linux source is affected by being unpacked and hashed on APFS, which presumably changes the resulting out path. I'm attempting to get access to a big-parallel builder so hopefully I'll be able to build it myself.

lilyball commented 1 year ago

@tstat I finally managed to build and deploy to my raspberry pi, and now it can't find bluetooth. I double-checked and the /boot/config.txt does indeed have dtparam=krnbt=on, but rfkill doesn't list bluetooth and bluetoothctl says there's no default controller. The only thing I omitted from your example was the bluez-tools package, since I'm not sure what that's for (and the source repo lists it as a beta and hasn't had any commits in almost 3 years), but that shouldn't affect anything AFAICT.

Do you have any idea what's wrong?

EDIT: I'm going to file an issue on your repo for this instead.

lloeki commented 1 year ago

For those looking to fix this when using nixos-hardware (linuxPackages_rpi4 kernel, no change to useGenerationDeviceTree), here's a quick hack to apply the above bits to the device tree Linux-side:

# bluetooth.nix

{ config, lib, pkgs, ... }:

let
  cfg = config.hardware.raspberry-pi."4".bluetooth;
in
{
  options.hardware = {
    raspberry-pi."4".bluetooth = {
      enable = lib.mkEnableOption ''
        configuration for bluetooth
      '';
    };
  };

  config = lib.mkIf cfg.enable {
    hardware.raspberry-pi."4".apply-overlays-dtmerge.enable = lib.mkDefault true;
    # doesn't work for the CM module, so we exclude e.g. bcm2711-rpi-cm4.dts
    hardware.deviceTree.filter = "bcm2711-rpi-4*.dtb";

    hardware.deviceTree = {
      overlays = [
        {
          name = "bluetooth-overlay";
          dtsText = ''
            /dts-v1/;
            /plugin/;

            / {
                compatible = "brcm,bcm2711";

                fragment@0 {
                    target = <&uart0_pins>;
                    __overlay__ {
                            brcm,pins = <30 31 32 33>;
                            brcm,pull = <2 0 0 2>;
                    };
                };
            };
          '';
        }
      ];
    };
  };
}
# configuration.nix

  imports =
    [
      # ...
      ./bluetooth.nix
      # ...
    ];

  hardware.bluetooth.enable = true;
  hardware.raspberry-pi."4".bluetooth.enable = true;

  systemd.services.btattach = {
    before = [ "bluetooth.service" ];
    after = [ "dev-ttyAMA0.device" ];
    wantedBy = [ "multi-user.target" ];
    serviceConfig = {
      ExecStart = "${pkgs.bluez}/bin/btattach -B /dev/ttyAMA0 -P bcm -S 3000000";
    };
  };

With that I was able to:

I'm probably going to contribute that back to the nixos-hardware repo.

n8henrie commented 1 year ago

I've been tinkering with this for the past few days. I'm using the rpi kernel (via nixos-hardware) on my Pi 4, in large part so I have access to /dev/gpiomem for some GPIO home automation stuff.

For my Pi4, which uses the nixos-hardware pi4 module (which sets the kernel to linuxPackages_rpi4), I did not have luck with:

I did have to use:

I would rather be using the mainline kernel, but then I don't know how to get /dev/gpiomem, so this config seems workable for now.

n8henrie commented 1 year ago

I'm on ZFS root and tried switching over to the uefi firmware and grub with kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; (6.1.45). I removed all references to nixos-hardware and all the deviceTree stuff from above, thinking that bluetooth might "just work", as this config seems pretty similar to what @pschyska was using above.

hci0 appears with rfkill list and is unblocked, but unfortunately I get No default controller available when I try to power on, and btattach just hangs. No difference with or without dtparam=krnbt=on.

I see the following in dmesg after btattach:

Bluetooth: hci1: command 0x0c03 tx timeout
Bluetooth: hci1: BCM: Reset failed (-110)

I also seem to only get 2 GPIO pins listed under /dev/gpiochip0, which seems weird:

$ sudo gpioinfo
gpiochip0 - 2 lines:
        line   0:      unnamed       unused  output  active-high
        line   1:      unnamed       unused  output  active-high
gpiochip1 - 3 lines:
        line   0:      unnamed       unused  output  active-high
        line   1:      unnamed       unused  output  active-high
        line   2:      unnamed       unused  output  active-high

Boot fails when I try to switch to the rpi kernel.

n8henrie commented 10 months ago

I feel bad that I never followed up above. My Pi4 has been running fairly well for the last several months with the following setup, which provided functional GPIO and bluetooth

Unfortunately I updated to 23.11 a few days ago and bluetooth is gone again.

EDIT: Bluetooth is working again on 23.11, I think the issue was with my UEFI setup, after reconfiguring that and a reboot I'm back in business.