Import Hetzner aarch64-linux builder config into delft/

[delroth]

So that everyone else in core-infra can access it, deploy changes, and fix issues.

Possibly blocked on #324?

[delroth]

(Alternatively, if you don't have the time to actually do the work integrating this into delft/, feel free to just dump the existing config here or in a gist or something, and someone else can take a look later.)

[vcunat]

It was a quick copy&paste of various bits:

{ config, pkgs, lib, ... }:

with (import ./common-let.nix pkgs);
{
  environment.systemPackages = with pkgs; [
    atop lm_sensors
    git fish
    # nvme smart-log -H /dev/nvme0
    smartmontools nvme-cli
  ];

  fileSystems = with fsOpts; {
    "/".options = btrfs_new ++ [ "commit=300" ];
  };

  hardware.enableRedistributableFirmware = true;
  #hardware.rasdaemon.enable = true; # ??

  networking = {
    firewall.allowPing = true;
  };

  #nix.package = pkgs.nixVersions.nix_2_13; # hope for GC with fewer errors
  nix.settings = {
    auto-optimise-store = true; # good on SSD
    experimental-features = "nix-command";
  };

  # ??
  # Better default than what's generated into ./hardware-configuration.nix
  powerManagement.cpuFreqGovernor = lib.mkForce "schedutil";

  services = {
    # TODO: ?
    #journald.extraConfig = "SystemMaxUse=1G";

    fstrim.enable = lib.mkDefault true; # good, probably?

    openssh = {
      enable = true;
      settings.PasswordAuthentication = false;
    };
  };

  time.timeZone = "Europe/Prague";

  users = {
    mutableUsers = false;
    extraUsers.root.openssh.authorizedKeys.keys = with sshKeys; [
      vcunat.zazen vcunat.nics vcunat.hime
      hydra.farm ''command="nix-store --serve --write" ${hydra.queue}''
    ];
  };

  zramSwap.enable = true;

  ### Various tweaks, from:
  ### https://github.com/NixOS/nixos-org-configurations/blob/master/delft/common.nix

  # Bump the open files limit so that non-root users can run NixOS VM
  # tests (Samba opens lot of files).
  security.pam.loginLimits =
    [ { domain = "*"; item = "nofile"; type = "-"; value = "16384"; }
    ];

  # Enable Kernel Samepage Merging (reduces memory footprint of VMs).
  systemd.services."enable-ksm" =
    { description = "Enable Kernel Same-Page Merging";
      wantedBy = [ "multi-user.target" ];
      after = [ "systemd-udev-settle.service" ];
      script =
        ''
          if [ -e /sys/kernel/mm/ksm ]; then
            echo 1 > /sys/kernel/mm/ksm/run
          fi
        '';
    };

  # Disable the systemd-journald watchdog. The default timeout (1min)
  # can easily be triggered on our slow, heavily-loaded disks. And
  # that may cause services writing to the journal to fail until
  # they're restarted.
  systemd.services.systemd-journald.serviceConfig.WatchdogSec = 0;

  ### Garbage collection   FIXME: improve? A custom script perhaps?
  # https://github.com/NixOS/nixos-org-configurations/blob/master/delft/build-machines-common.nix
  # Decreased interval, as the collected amounts tended to be rather low.
  nix.gc = {
    automatic = true;
    dates = "03,15:15";
    randomizedDelaySec = "1h"; # don't block all build machines at the same time.
  };
}

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "nvme" "xhci_pci" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/0f8cb540-99f0-4b71-8206-0fb813c2b0b0";
      fsType = "btrfs";
    };

  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/28C3-5AC2";
      fsType = "vfat";
    };

  swapDevices = [ ];

  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
  # (the default) this is the recommended approach. When using systemd-networkd it's
  # still possible to use this option, but it's recommended to use it in conjunction
  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
  networking.useDHCP = lib.mkDefault true;
  # networking.interfaces.eno1.useDHCP = lib.mkDefault true;
  # networking.interfaces.eno2.useDHCP = lib.mkDefault true;

  nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux";
}

[vcunat]

I'm not sure if there's much content worth salvaging.

[mweinelt]

By the way, there is an option for KSM: hardware.ksm.enable.

It was imported from this repo 9 years ago into nixpkgs :laughing: https://github.com/NixOS/nixpkgs/commit/224ed7e798caf80e65250385a9cb0ab09c52d2c7

[delroth]

Other option that @mweinelt suggested: we could make the Hetzner aarch64-linux builder boot from a similar iPXE image to what Packet uses, and make it netboot using the same image.

Sounds like a really good idea to me, actually.

[delroth]

Filed a support ticket to get a USB drive attached to the server so we can put on it:

• An iPXE build with embedded script.
• Secrets like the SSH host key (since we can't broker it via the Equinix Metal API).
• Possibly: network config data since we don't have a DHCP.