Setup NixOS Hypervisor

[mvgijssel]

closes #628

TODO

• [x] Disable bluetooth tracking in esphome devices
• [x] Flash NixOS image to USB (https://nixos.org/manual/nixos/stable/#sec-booting-from-usb)
• [x] Install NixOS on Hypervisor machine
• [x] Configure local SSH key and store in 1Password
• [x] Prevent password logins SSH
• [x] Prevent root login SSH (https://nixos.wiki/wiki/SSH_public_key_authentication)
• [x] Configure ZFS (https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/index.html#installation)
• [x] Configure static network ip
• [x] Configure K3S
• [x] Install Kubevirt
• [x] Install HAOS virtual machine
• [x] Symlink cni directories of k3s to the right place
• [x] Fix /etc/localtime time of the container and schedule of the lights
• [x] Make mutagen works with nixos host
• [x] Ability to deploy hypervisor from macOS

[mvgijssel]

Trying to install NixOS:

• First USB didn't boot to the NixOS installer - installed by Etcher
• Second USB did boot to NixOS to kernel panic immediately after - installed by Etcher
• Third USB boot to NixOS installer without a problem - installed using sudo dd if=/Users/maarten/Downloads/nixos-minimal-23.11.2130.d65bceaee0fb-x86_64-linux.iso of=/dev/rdisk4 bs=4M

[mvgijssel]

Import the zpool:

zpool import
# find the proper pool name
zpool import -f new_data
zpool status
zpool upgrade
zpool update new_data

[mvgijssel]

Installing kubevirt from https://kubevirt.io/quickstart_cloud/

export VERSION=$(curl -s https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
echo $VERSION
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml
kubectl get all -n kubevirt # poll until ready
export VERSION=$(basename $(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest))
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
kubectl get cdi cdi -n cdi # poll until done
kubectl get all -n cdi # poll until done

From https://github.com/kubevirt/containerized-data-importer/issues/3005

kubectl patch --type merge -p '{"spec": {"claimPropertySets": [{"accessModes": ["ReadWriteOnce"]}]}}' StorageProfile local-path

Setup HAOS using (https://charlottemach.com/2020/11/03/windows-kubevirt-k3s.html)

cat <<EOF > dv_haos.yml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "haos"
spec:
  storage:
    resources:
      requests:
        storage: 40Gi
  source:
    http:
      url: "https://github.com/home-assistant/operating-system/releases/download/11.2/haos_ova-11.2.qcow2.xz"
EOF

kubectl create -f dv_haos.yml

cat <<EOF > nad_haos.yml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: bridge-network
spec:
  config: '{
    "cniVersion": "1.0.0",
    "name": "bridge-network",
    "type": "bridge",
    "bridge": "br0"
}'
EOF

kubectl create -f nad_haos.yml

cat <<EOF > vm_haos.yml
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  creationTimestamp: 2018-07-04T15:03:08Z
  generation: 1
  labels:
    kubevirt.io/os: linux
  name: haos
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: haos
    spec:
      domain:
        cpu:
          cores: 2
        devices:
          disks:
            - disk:
                bus: virtio
              name: disk0
          interfaces:
            - name: default
               masquerade: {}
            - name: bridge-network
              bridge: {}
        machine:
          type: q35
        resources:
          requests:
            memory: 4096M
      networks:
        - name: bridge-network
          multus:
            networkName: bridge-network
        - name: default
           pod: {} # Stock pod network
      volumes:
        - name: disk0
          persistentVolumeClaim:
            claimName: haos
EOF

kubectl create -f vm_haos.yml

kubectl get pods # wait for the importer process to complete
kubectl proxy --address=0.0.0.0 --accept-hosts='^*$' --port 8080 # port forward 8080 to localhost
virtctl console haos

[mvgijssel]

k3s has cni in non-default location, so we need to symlink directories for multus to work:

/var/lib/rancher/k3s/agent/etc/cni/net.d -> /etc/cni/net.d
/var/lib/rancher/k3s/data/current/bin -> /opt/cni/bin

do this using article https://www.reddit.com/r/NixOS/comments/ckg1sr/how_can_i_turn_a_path_to_stateful/

systemd.tmpfiles.rules = [ "d /var/lib/foobar 0750 wwwrun wwwrun - -" "d /var/cache/foobar 0750 wwwrun wwwrun - -" ];

Just need to know how to specify a symlink

[mvgijssel]

Installing multus

kubectl create -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml
kubectl get pods --all-namespaces | grep -i multus
ls -la /etc/cni/net.d # check for multus here
ls -la /opt/cni/bin # check for multus here
kubectl get net-attach-def --all-namespaces
journalctl -u k3s

[mvgijssel]

PIVOT: install docker and boot the legacy setup using docker compose. That way we can slowly figure out how to run HAOS with Kubevirt and can migrate all components one by one without downtime.

[mvgijssel]

Using https://cln.io/blog/homeassistant-relink-homekit-bridge-to-homekit-again/#erasing-wiping-the-homekit-bridge-state-file-from-home-assistant repair the Homekit and Home Assistant bridge integration.

[mvgijssel]

Make sure to update the NixOS firewall settings to enable Homekit bridge to work.

[mvgijssel]

Can use the following from macOS:

NIX_SSHOPTS="-o ForwardAgent=yes" nixos-rebuild switch -I nixos-config=configuration.nix --target-host maarten@192.168.1.30 --build-host maarten@192.168.1.30 --fast --use-remote-sudo