Can't deploy to NixOS

[jafonsor]

Hey :wave: Can somebody help me?

I am new to nix and NixOs. I was following a tutorial for delpoying an haskell web server using nixops. I diverged from the tutorial, because I want to deploy to a NixOs system instead of the AWS.

By reading the manual I figured I could deploy to a nix just by setting the deployment.targetHost to the IP of the server (which by the way is running on my local network).

However I got this error:

~/nixops-example$ nixops create ./nixops-example-configuration.nix ./nixops-example-lan.nix -d nixops-example
created deployment ‘9b1e9f17-c51a-11e9-a315-2f6bb2201f20’
9b1e9f17-c51a-11e9-a315-2f6bb2201f20

~/nixops-example$ nixops list
+--------------------------------------+----------------+------------------------+------------+------+
| UUID                                 | Name           | Description            | # Machines | Type |
+--------------------------------------+----------------+------------------------+------------+------+
| 9b1e9f17-c51a-11e9-a315-2f6bb2201f20 | nixops-example | Unnamed NixOps network |          0 |      |
+--------------------------------------+----------------+------------------------+------------+------+

~/nixops-example$ nixops deploy -d nixops-example
nixops-example> generating new SSH keypair... done
nixops-example> setting state version to 19.03
nixops-example> waiting for SSH...
building all machine configurations...
error: 
Failed assertions:
- The ‘fileSystems’ option does not specify your root file system.
- You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable.
(use '--show-trace' to show detailed location information)
error: unable to build all machine configurations

nixops-example-configuration.nix:

{
  network.description = "nixops example";

  nixops-example =
    {config, pkgs, ...}:
      let
        nixops-example = import ./nixops-example/default.nix { inherit pkgs; };
      in
        {
          networking.hostName = "nixops example";
          networking.firewall.allowedTCPPorts = [ 22 29051 ];
          environment.systemPackages = [ nixops-example ];
          systemd.services.nixops-example = {
            description = "nixops example of web server deploy";
            wantedBy = [ "multi-user.target" ];
            after = [ "network.target" ];
            serviceConfig = {
              ExecStart = "${nixops-example}/bin/nixops-example";
            };
          };
        };

}

nixops-example-lan.nix:

{
  nixops-example =
    { config, pkgs, ... }: {
      deployment.targetHost = "192.168.1.200";
    };
}

The full code is on this repository.

[asymmetric]

Have you tried doing what the error message says, i.e. setting boot.loader.grub.devices?

[jafonsor]

Thanks for your answer. I am assuming the error is about the/etc/nixos/configuration.nix in the machine that is the target of the deploy. The field is already set (boot.loader.grub.device = "/dev/sda";). I don't know much about nix. I think that according to the this comment about the boot.loader.grub.device on the grub package grub.nix package It should be enough:

         The device on which the GRUB boot loader will be installed.
          The special value <literal>nodev</literal> means that a GRUB
          boot menu will be generated, but GRUB itself will not
          actually be installed.  To install GRUB on multiple devices,
          use <literal>boot.loader.grub.devices</literal>.

Here is the full /etc/nixos/configuration.nix on the target of the deploy. The machine I'm trying to deploy from is running ubuntu.

# Edit this configuration file to define what should be installed on
# your system.  Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).

{ config, pkgs, ... }:

{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];

  nixpkgs.config.allowUnfree = true;
  boot.kernelPackages = pkgs.linuxPackages_latest;

  # Use the GRUB 2 boot loader.
  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  # boot.loader.grub.efiSupport = true;
  # boot.loader.grub.efiInstallAsRemovable = true;
  # boot.loader.efi.efiSysMountPoint = "/boot/efi";
  # Define on which hard drive you want to install Grub.
  boot.loader.grub.device = "/dev/sda"; # "/dev/sda"; # or "nodev" for efi only
  boot.loader.grub.useOSProber = true;

  # networking.hostName = "nixos"; # Define your hostname.
  # networking.wireless.enable = true;  # Enables wireless support via wpa_supplicant.
  # networking.networkmanager.enable = true;
  networking = {
    usePredictableInterfaceNames = false;
    interfaces.eth0.ipv4.addresses = [{
      address = "192.168.1.200";
      prefixLength = 24;
    }];
    defaultGateway = "192.168.1.1";
    nameservers = [ "8.8.8.8" ];
  };

  # Configure network proxy if necessary
  # networking.proxy.default = "http://user:password@proxy:port/";
  # networking.proxy.noProxy = "127.0.0.1,localhost,internal.domain";

  # Select internationalisation properties.
  i18n = {
  #   consoleFont = "Lat2-Terminus16";
    consoleKeyMap = "pt-latin1";
  #   defaultLocale = "en_US.UTF-8";
  };

  # Set your time zone.
  time.timeZone = "Europe/Lisbon";

  # List packages installed in system profile. To search, run:
  # $ nix search wget
  environment.systemPackages = with pkgs; [
     wget vim docker-compose htop
  ];

  # Some programs need SUID wrappers, can be configured further or are
  # started in user sessions.
  # programs.mtr.enable = true;
  # programs.gnupg.agent = { enable = true; enableSSHSupport = true; };

  # List services that you want to enable:

  # Enable the OpenSSH daemon.
  services.openssh.enable = true;
  services.openssh.permitRootLogin = "yes";
  services.openssh.authorizedKeysFiles = ["~/.ssh/authorized_keys"];

  # Open ports in the firewall.
  # networking.firewall.allowedTCPPorts = [ ... ];
  # networking.firewall.allowedUDPPorts = [ ... ];
  # Or disable the firewall altogether.
  # networking.firewall.enable = false;

  # Enable CUPS to print documents.
  # services.printing.enable = true;

  # Enable sound.
  # sound.enable = true;
  # hardware.pulseaudio.enable = true;

  # Enable the X11 windowing system.
  # services.xserver.enable = true;
  # services.xserver.layout = "us";
  # services.xserver.xkbOptions = "eurosign:e";

  # Enable touchpad support.
  # services.xserver.libinput.enable = true;

  # Enable the KDE Desktop Environment.
  # services.xserver.displayManager.sddm.enable = true;
  # services.xserver.desktopManager.plasma5.enable = true;

  # Define a user account. Don't forget to set a password with ‘passwd’.
  # users.users.guest = {
  #   isNormalUser = true;
  #   uid = 1000;
  # };
  users.users.joao = {
    isNormalUser = true;
    home = "/home/joao";
    extraGroups = [ "wheel" "networkmanager" "docker" ];
  };

  # This value determines the NixOS release with which your system is to be
  # compatible, in order to avoid breaking some software such as database
  # servers. You should change this only after NixOS release notes say you
  # should.
  system.stateVersion = "18.09"; # Did you read the comment?

  virtualisation.docker.enable = true;
}

[jafonsor]

@asymmetric, Where should I set the ‘fileSystems’?

I'm trying to figure out a solution. I ran the command with --show-trace and got this:

building all machine configurations...
error: while evaluating the attribute 'buildCommand' of the derivation 'nixops-machines' at /nix/store/xyi151lmvfwzpw8mcb2qdiar3vggam93-nixpkgs-18.09beta609.3768913cc9e/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:177:11:
while evaluating anonymous function at /nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/share/nix/nixops/eval-machine-info.nix:368:46, called from undefined position:
while evaluating the attribute 'config.system.build.toplevel' at /home/joao/.nix-defexpr/channels/nixpkgs/nixos/modules/system/activation/top-level.nix:272:5:
while evaluating 'foldr' at /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:36:20, called from /home/joao/.nix-defexpr/channels/nixpkgs/nixos/modules/system/activation/top-level.nix:136:12:
while evaluating 'fold'' at /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:39:15, called from /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:43:8:
while evaluating 'showWarnings' at /home/joao/.nix-defexpr/channels/nixpkgs/nixos/modules/system/activation/top-level.nix:100:18, called from /home/joao/.nix-defexpr/channels/nixpkgs/nixos/modules/system/activation/top-level.nix:107:16:
while evaluating 'foldr' at /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:36:20, called from /home/joao/.nix-defexpr/channels/nixpkgs/nixos/modules/system/activation/top-level.nix:100:23:
while evaluating 'fold'' at /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:39:15, called from /home/joao/.nix-defexpr/channels/nixpkgs/lib/lists.nix:43:8:

Failed assertions:
- The ‘fileSystems’ option does not specify your root file system.
- You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable.
Traceback (most recent call last):
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/bin/..nixops-wrapped-wrapped", line 985, in <module>
    args.op()
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/bin/..nixops-wrapped-wrapped", line 407, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate)
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/lib/python2.7/site-packages/nixops/deployment.py", line 1051, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/lib/python2.7/site-packages/nixops/deployment.py", line 1040, in run_with_notify
    f()
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/lib/python2.7/site-packages/nixops/deployment.py", line 1051, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/lib/python2.7/site-packages/nixops/deployment.py", line 991, in _deploy
    self.configs_path = self.build_configs(dry_run=dry_run, repair=repair, include=include, exclude=exclude)
  File "/nix/store/amp5hys6c41hzhzjy5k69q9qb8wkj44n-nixops-1.6/lib/python2.7/site-packages/nixops/deployment.py", line 659, in build_configs
    raise Exception("unable to build all machine configurations")
Exception: unable to build all machine configurations

I figured that the error is printed by the command:

nix-build 
 -I nixops=/nix/store/zrkm7my0n7j4w8lh6aqmsflf1wqmmwip-nixops-1.7pre0_abcdef/share/nix/nixops 
 --arg networkExprs '[ "/home/joao/progs/obelisk/nixops-example/nixops-example-configuration.nix" "/home/joao/progs/obelisk/nixops-example/nixops-example-lan.nix" "/tmp/nixops-tmpP1YvKL/physical.nix" ]'
 --arg args {} 
 --argstr uuid '9b1e9f17-c51a-11e9-a315-2f6bb2201f20' 
 --argstr deploymentName 'nixops-example' <nixops/eval-machine-info.nix> 
 --arg names '[ "nixops-example" ]'
 -A machines 
 -o /tmp/nixops-tmpP1YvKL/configs

I can't run it standalone because the file /tmp/nixops-tmpP1YvKL/physical.nix that I think is generated before the command is ran.

[jafonsor]

I guess fileSystems is set on /etc/nixos/hardware-configuration.nix:

[root@nixos:/etc/nixos]$ cat hardware-configuration.nix 
# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, ... }:

{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];

  boot.initrd.availableKernelModules = [ "uhci_hcd" "ahci" "usbhid" ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/1142ac37-0fa1-44b6-9bc5-5fa034eb8862";
      fsType = "ext4";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/bc593993-02b9-4110-b483-bc5248edee3e";
      fsType = "ext4";
    };

  swapDevices =
    [ { device = "/dev/disk/by-uuid/5ac8ddaa-e5f6-47d7-b838-da4216397475"; }
    ];

  nix.maxJobs = lib.mkDefault 4;
  powerManagement.cpuFreqGovernor = lib.mkDefault "ondemand";
}

[aij]

@jafonsor Yes. When deployed, the NixOps configuration for that machine will be the whole configuration for that machine, replacing the NixOS config that was originally used to install it. So you'll want the hardware configuration and any other required configuration to be set in your NixOps config.

The easiest way I've found to manage that is to copy hardware-configuration.nix and a minimal version of configuration.nix and import it into the NixOps config for the corresponding machine. (I keep them in a git submodule, but keeping them in the same repo could also make sense.)

[jafonsor]

Oh boy. I was totally clueless. Thanks for your help. Just out of curiosity, is it possible to deploy two services to the same NixOS machine without having them to know about each other? Who would you run several web servers on the same NixOS machine?

Where should I set the hardware-configuration.nix and the configuration.nix in the NixOps configuration? I think the manual could use an update, since for NixOS deployments it just says:

To deploy to a machine that is already running NixOS, simply set deployment.targetHost to the IP address or host name of the machine, and leave deployment.targetEnv undefined.

[lordcirth]

@jafonsor Normally you don't write configuration.nix via Nixops; you just put the relevant options in your Nixops config. The bare minimum of a / filesystem and grub should work.

[toraritte]

@jafonsor I think this is what @aij was talking about (a search in his aij/aij-nixos-config repo):

https://github.com/aij/aij-nixos-config/blob/master/machines/m0/default.nix

{ config, lib, pkgs, nodes, ... }: {
  # Cisco C240, 2x E5-2650v2
  networking.hostName = "m0"; # Formerly "c240-0";
  networking.hostId = "a2853a4d";
  system.stateVersion = "18.03";

  imports =
    [ ./hardware-configuration.nix
      ../../nixops.nix
    ];

  boot.loader.grub = {
    enable = true;
    version = 2;
    efiSupport = true;
    #efiInstallAsRemovable = true;
    device = "nodev";
    #device = "/dev/disk/by-id/ata-ST4000LM024-2AN17V_WCK0WHYR-part2";
  };
}

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-images-on-azure/7062/11

[lava]

I also ran into this issue and I'm a bit confused by the current status. Does this mean that the example in the manual (https://releases.nixos.org/nixops/nixops-1.7/manual/manual.html#ex-physical-nixos.nix) has never worked at all? Or was this broken more recently because NixOS added additional assertions?

Also, is deploying to NixOS something that's still supported and a config in the style of the one linked by @toraritte is the recommended way of doing it, or are these workarounds and the "intended" way of using nixops is to use one of the other backends to spin up a new machine for a deployment?

[aij]

@lava It's still working for me, though if it makes a difference I'm using nixopsUnstable. IIRC there was something broken in nixops 1.x that made it unusable for me, but I forget the details.

I can't say whether this is the way it was intended to be used, but it's what I was led to do... Nixops deploys a full system configuration and AFAICT has no way to fetch details from a bare metal target, so you end up needing to define the full system config locally.