"Did not find a cmdline Flattened Device Tree"

[sergueif]

Followed the steps, built an sd image put one the example configuration.nix (with edits suggested by comments) from README.md for rPi3 into /etc/nixos/configuration.nix. Ran nixos-rebuild switch Returns successfully and new packages are even there. But the system doesn't survive reboot (shutdown -r now) Once booted the next time, using the default or latest configuration results in

ERROR: Did not find a cmdline Flattened Device Tree

and when trying to use the old configuration "1" , it gets to something like "Starting Kernel...." and just hangs there

Tried a few tweaks to configuration.nix thinking that's the problem but with no success.

Is it something wrong about how I'm making the SD card? I even gave it the "NIXOS_SD" label. But should I format it some special way with Disk Utility? (I'm on mac) I don't have a way to format the SD with ext4 but does it even matter? i.e. doesn't dd if=.... command put it's own file system and all?

Also, I notice the built version of the NixOS is something ....pre-git. Is there a "stable" version I can build instead?

What am I missing?

Here's my configuration.nix if it's useful:

# Please read the comments!
{ config, pkgs, lib, ... }:
{
  # Boot
  boot.loader.grub.enable = false;
  boot.loader.raspberryPi.enable = true;
  boot.loader.raspberryPi.version = 3;
  boot.loader.raspberryPi.uboot.enable = true;

  # Kernel configuration
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = ["cma=32M"];

  # Enable additional firmware (such as Wi-Fi drivers).
  hardware.enableRedistributableFirmware = true;

  # Filesystems
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  # Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
  networking.hostName = "nixpi"; # unleash your creativity!

  # Packages
  environment.systemPackages = with pkgs; [
    # customize as needed!
    vim git htop nethack wget
  ];

  # Users
  users.users.nixie = {
    isNormalUser = true;
    # Don't forget to change the home directory too.
    home = "/home/nixie";
    # This allows this user to use `sudo`.
    extraGroups = [ "wheel" ];
    # SSH authorized keys for this user.
    openssh.authorizedKeys.keys = [ "mah ssh key..." ];
  };

  # Miscellaneous
  time.timeZone = "America/Toronto"; # you probably want to change this -- otherwise, ciao!
  services.openssh.enable = true;

  # WARNING: if you remove this, then you need to assign a password to your user, otherwise
  # `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
  # by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
  security.sudo.wheelNeedsPassword = false;

  # Nix
  nix.gc.automatic = true;
  nix.gc.options = "--delete-older-than 30d";
  boot.cleanTmpDir = true;

  # https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
  system.stateVersion = "20.03";
}

[Robertof]

Hi there!

Interesting message - that error is caused by UBoot. Does /boot look alright?

What did you use to flash the SD image? dd should suffice and produce a working result - some people had trouble with Etcher on macOS in the past.

Regarding the pre-git, that shouldn't matter as long as di you didn't change the NIXPKGS_BRANCH option in docker-compose.yml:

https://github.com/Robertof/nixos-docker-sd-image-builder/blob/03359cbe7114901abe4f935fc1e633d9c8746b8f/docker/docker-compose.yml#L17

[sergueif]

How can i check /boot after it gets into this state? Cause, none of the configurations boot (even though in different way) I'm able to pause though, before running ‘boot’ and getting a list of configurations?

Sent from my iPhone

On Feb 13, 2021, at 12:50, Roberto Frenna notifications@github.com wrote:

 Hi there!

Interesting message - that error is caused by UBoot. Does /boot look alright?

What did you use to flash the SD image? dd should suffice and produce a working result - some people had trouble with Etcher on macOS in the past.

Regarding the pre-git, that shouldn't matter as long as di you didn't change the NIXPKGS_BRANCH option in docker-compose.yml:

https://github.com/Robertof/nixos-docker-sd-image-builder/blob/03359cbe7114901abe4f935fc1e633d9c8746b8f/docker/docker-compose.yml#L17

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[Robertof]

You can check just before rebooting and after nixos-rebuild switch! In my case:

# ls -l /boot/
total 12536
-r--r--r-- 1 root root   52296 Jan 10 12:41 bootcode.bin
-r--r--r-- 1 root root     374 Jan 10 12:41 config.txt
drwxr-xr-x 2 root root    4096 Jan 10 12:41 extlinux
-r--r--r-- 1 root root    2657 Jan 10 12:41 fixup_cd.dat
-r--r--r-- 1 root root    6736 Jan 10 12:41 fixup.dat
-r--r--r-- 1 root root    9808 Jan 10 12:41 fixup_db.dat
-r--r--r-- 1 root root    9810 Jan 10 12:41 fixup_x.dat
drwxr-xr-x 3 root root    4096 May 10  2020 nixos
-r--r--r-- 1 root root  685668 Jan 10 12:41 start_cd.elf
-r--r--r-- 1 root root 4854728 Jan 10 12:41 start_db.elf
-r--r--r-- 1 root root 2877988 Jan 10 12:41 start.elf
-r--r--r-- 1 root root 3792232 Jan 10 12:41 start_x.elf
-r-xr-xr-x 1 root root  512696 Jan 10 12:41 u-boot-rpi.bin

I'm still on NixOS 20.03 though and it's been a while since the last update (as you can see by the timestamps), so YMMV!

[sergueif]

I'll give that a try. For the moment, I switched to trying out NixOS on an old thinkpad laptop without using this image build. I also can connect tv and keyboard to the 'pi' so the custom image isn't strictly necessary for me. Can I skip some steps if that's the case. I tried getting the aarch64 image directly but "hydra.nixos.org" was down for a while.

Anyway, I might give this method here another time soon

[Robertof]

Excellent - thanks! There's absolutely no need to create a custom image if you have the chance to remotely interact with the Pi. Also, Hydra seems to be back up now!

[sergueif]

Hi again, (warning: lots of pasted config incoming at the end but still readable)

I tried again, same result, but this time I took notes. Some things I'm confused by: 1) the tutorial calls to invoke nixos-generate-config but the "Example configuration for Pi 3" ignores it completely (overwrites configuration.nix and doesn't import hardware-configuration.nix 2) sudo blkid lists 2 partitions on the sd card with the label NIXOS_SD. Is that right? I use dd to write the sd card. which partition should I use? (see below blkid and fdisk -l output 3) generated "hardware" config sets the / fileSystem to a tiny 30M partition1, but recommended config sets / to NIXOS_SD, but there's 2 of them on the card.

Bottom line: I think, in my attempt to "combine" the generated and recommended config (from readme) is not working well. Or the partitions on the SD card aren't produced right. The tutorial should make clear what to keep from generated config or what to blow away. And how to confirm on first boot that the SD card is in right shape blkid / fdisk

before nixos-rebuild switch

**\[nixos@nixos:~\]$** ls -l /boot/
total 8
drwxr-xr-x 2 root root 4096 Jan 1 1970 **extlinux**
drwxr-xr-x 3 root root 4096 Jan 1 1970 **nixos**

blkid

[nixos@nixos:~]$ sudo blkid 
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL_FATBOOT="NIXOS_SD" LABEL="NIXOS_SD" UUID="2178-694E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="2178694e-01"
/dev/mmcblk0p2: LABEL="NIXOS_SD" UUID="44444444-4444-4444-8888-888888888888" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="2178694e-02"

fdisk -l

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1      16384    77823    61440   30M  b W95 FAT32
/dev/mmcblk0p2 *    77824 60367871 60290048 28.7G 83 Linux

generated hardware-configuration.nix

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "usbhid" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/2178-694E";
      fsType = "ext4";
    };

  swapDevices = [ ];

  powerManagement.cpuFreqGovernor = lib.mkDefault "ondemand";
}

generated configuration.nix

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  # Use the extlinux boot loader. (NixOS wants to enable GRUB by default)
  boot.loader.grub.enable = false;
  # Enables the generation of /boot/extlinux/extlinux.conf
  boot.loader.generic-extlinux-compatible.enable = true;

  # networking.hostName = "nixos"; # Define your hostname.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.

  # Set your time zone.
  # time.timeZone = "Europe/Amsterdam";

  # The global useDHCP flag is deprecated, therefore explicitly set to false here.
  # Per-interface useDHCP will be mandatory in the future, so this generated config
  # replicates the default behaviour.
  networking.useDHCP = false;
  networking.interfaces.eth0.useDHCP = true;
  networking.interfaces.wlan0.useDHCP = true;

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Select internationalisation properties.
  # i18n.defaultLocale = "en_US.UTF-8";
  # console = {
  #   font = "Lat2-Terminus16";
  #   keyMap = "us";
  # };

  # Configure keymap in X11
  # services.xserver.layout = "us";
  # services.xserver.xkbOptions = "eurosign:e";

  # Enable CUPS to print documents.
  # services.printing.enable = true;

  # Enable sound.
  # sound.enable = true;
  # hardware.pulseaudio.enable = true;

  # Enable touchpad support (enabled default in most desktopManager).
  # services.xserver.libinput.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  # users.users.jane = {
  #   isNormalUser = true;
  #   extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  # };

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  # environment.systemPackages = with pkgs; [
  #   wget vim
  #   firefox
  # ];

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = {
  #   enable = true;
  #   enableSSHSupport = true;
  # };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  # services.openssh.enable = true;

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "20.09"; # Did you read the comment?

}

attempt at combining generated and suggested

[nixos@nixos:~]$ cat /etc/nixos/configuration.nix
{ config, pkgs, ... }:
{
  imports =
    [ ./hardware-configuration.nix ];

  boot.loader.grub.enable = false;
  boot.loader.generic-extlinux-compatible.enable = true;
  boot.loader.raspberryPi.enable = true;
  boot.loader.raspberryPi.version = 3;
  boot.loader.raspberryPi.uboot.enable = true;
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = ["cma=32M"];
  boot.cleanTmpDir = true;

  hardware.enableRedistributableFirmware = true;

  networking.hostName = "nixpi";
  networking.useDHCP = false;
  networking.interfaces.eth0.useDHCP = true;
  networking.interfaces.wlan0.useDHCP = true;

  users.users.nixie = {
    isNormalUser = true;
    home = "/home/nixie";
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [ "...mah key..." ];
  };

  security.sudo.wheelNeedsPassword = false;

  time.timeZone = "America/Toronto";

  services.openssh.enable = true;
  services.xserver.layout = "us";
  services.xserver.libinput.enable = true;

  nix.gc.automatic = true;
  nix.gc.options = "--delete-older-than 30d";

  i18n.defaultLocale = "en_US.UTF-8";

  console = {
    font = "Lat2-Terminus16";
    keyMap = "us";
  };

  sound.enable = true;

  system.stateVersion = "20.09";

}

boot after nixos-rebuild switch

[nixos@nixos:~]$ ls -l /boot/
total 8
drwxr-xr-x 2 root root 4096 Feb 23 07:54 extlinux
drwxr-xr-x 4 root root 4096 Feb 23 07:54 nixos

[Robertof]

Hey!

Sorry for the late answer. Apologies for the confusion in the README - I'll try to improve the wording. You don't need to run nixos-generate-config at all if you're using the base config suggestion that I attached in the README, everything should work using just that!

In your case, something like this in /etc/nixos/configuration.nix (without any other file) should just work:

# Please read the comments!
{ config, pkgs, lib, ... }:
{
  # Boot
  boot.loader.grub.enable = false;
  boot.loader.raspberryPi.enable = true;
  boot.loader.raspberryPi.version = 3;
  boot.loader.raspberryPi.uboot.enable = true;

  # Kernel configuration
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.kernelParams = ["cma=32M"];

  # Enable additional firmware (such as Wi-Fi drivers).
  hardware.enableRedistributableFirmware = true;

  # Filesystems
  fileSystems = {
    "/" = {
      device = "/dev/disk/by-label/NIXOS_SD";
      fsType = "ext4";
    };
  };
  swapDevices = [ { device = "/swapfile"; size = 1024; } ];

  # Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
  networking.hostName = "nixpi";

  # Users
  users.users.nixie= {
    isNormalUser = true;
    # Don't forget to change the home directory too.
    home = "/home/nixie";
    # This allows this user to use `sudo`.
    extraGroups = [ "wheel" ];
    # SSH authorized keys for this user.
    openssh.authorizedKeys.keys = [ "... your key ..." ];
  };

  # Miscellaneous
  time.timeZone = "America/Toronto";
  services.openssh.enable = true;

  # WARNING: if you remove this, then you need to assign a password to your user, otherwise
  # `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
  # by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
  security.sudo.wheelNeedsPassword = false;

  # Nix
  nix.gc.automatic = true;
  nix.gc.options = "--delete-older-than 30d";
  boot.cleanTmpDir = true;

  # https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
  system.stateVersion = "20.09";
}

[sergueif]

trying this right now...

So it's not weird that the sd card has 2 partitions, 1 tiny, 1 big?

[nixos@nixos:~]$ sudo blkid 
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL_FATBOOT="NIXOS_SD" LABEL="NIXOS_SD" UUID="2178-694E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="2178694e-01"
/dev/mmcblk0p2: LABEL="NIXOS_SD" UUID="44444444-4444-4444-8888-888888888888" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="2178694e-02"

fdisk -l
Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1      16384    77823    61440   30M  b W95 FAT32
/dev/mmcblk0p2 *    77824 60367871 60290048 28.7G 83 Linux

[sergueif]

damn, still get that "Did not find a cmdline Flattened Device Tree" after a nixos-rebuild switch and sudo reboot

Maybe it's not the configuration.nix that's the problem, but something about the sd card or partitions. (see comment above) I'm using a real simple dd like:

sudo dd if=nixos-sd-image-20.09pre-git-aarch64-linux.img of=/dev/disk2 bs=64k

on mac os x. I usually doesn't put the "label" on the SD either. I rename the sd to NIXOS_SD using Disk Utility after dd.

[Robertof]

You shouldn't need to change any label on the SD card, that could actually cause the issue - the '.img' file already contains a partition with the correct label for NixOS to find. It's also normal to have two partitions, the first one contains the cmdline.txt file that the Raspberry Pi bootloader reads to understand what it needs to do and the other one contains the system. It should really work out of the box 🤔

[Robertof]

(I've flashed it before on macOS with coreutils installed via Brew and something like sudo gdd if=<image> of=/dev/rdiskN bs=4M oflag=sync)

[sergueif]

Thank you Roberof for helping me out.

I undestand now that "NIXOS_SD" should be label of the bigger partition and "FIRMWARE" is what I end up with as the label for the tiny first partition. So it makes sense that "root" of nixos is the bigger partition.

The outcome is still largely the same, but I have a slight improvement to report. So, after nixos-rebuild switch, the new configuration still doesn't boot and prints the "Did not find a cmdline Flattened Device Tree" on the TV connected to it. But if I unplug, plug power, interrupt boot and choose the older / original configuration, it prints something nice about finding a device tree, doesn't give me a login shell (even though after fresh dd it did), BUT...after a few seconds, I can SSH into the original SD image. So maybe that means there's some configuration tweaking I can do without having to re-flash the SD card every time it's unsuccessful.

Anything I can print output of here? I can sit in the sd-image configuration, I can get into UBoot console.

[sergueif]

progress! I think it works (or at least much better) after this change to the configuration (not sure which of the 3 lines made the difference):

  # Kernel configuration
  # boot.kernelPackages = pkgs.linuxPackages_latest;
  # boot.kernelParams = ["cma=32M"];

  #addition on op of Robertof
  boot.loader.generic-extlinux-compatible.enable = true;
  boot.kernelPackages = pkgs.linuxPackages_5_4;
  boot.kernelParams = ["cma=64M"];

I lifted that from some random config on the internet https://github.com/lucernae/nixos-pi/blob/main/configuration.nix

I still wish it gave me a shell on the TV+keyboard, but after each sudo reboot, I CAN ssh into it and it seems to work more or less like the Nixos I'm running on a bigger PC and thinkpad elsewhere.

[Robertof]

Hey! Very happy that something is starting to work for you!

Unfortunately I'm not nearly knowledgeable enough about UBoot to provide detailed debugging steps, however, for the console, perhaps you can try adding "console=tty0" to the kernel params, like:

boot.kernelParams = ["cma=64M" "console=tty0"];

Sorry again about all the trouble!

[sergueif]

Well, I think what this whole experiment contributes here, is one of these 3 lines made the difference to remedy the "Did not find a cmdline Flattened Device Tree" error:

#addition on top of Robertof
  boot.loader.generic-extlinux-compatible.enable = true;
  boot.kernelPackages = pkgs.linuxPackages_5_4;
  boot.kernelParams = ["cma=64M"];

Not sure what's the appropriate place in the readme to add that to.

[Robertof]

I've clarified the wording in the README file to make sure that there is no ambiguity around nixos-generate-config, and also added a link to this issue in the "Troubleshooting" section. Since you resolved the problem I'm going to close this issue - feel free to reopen in case anything else breaks!