convert 'proxmox' node from proxmox to truenas

billimek commented 3 years ago

Currently the host proxmox acts as both a NSF as well as a proxmox node (with an associated k3s VM).

It is not ideal to have the NAS act as double-duty as another purpose, especially in this environment with about 8 other physical nodes.

Therefore, the purpose of this issue to is facilitate the conversion of this host from a proxmox node to a standalone NAS, using the TrueNAS solution.

billimek commented 3 years ago

Workloads that will need to be addressed:

[x] NFS clients (see /etc/exports)
[x] borgbackup 'server' and associated configuration (see /etc/borgmatic/config.yaml, /etc/systemd/system/borgmatic.service, /etc/systemd/system/borgmatic.timer)
[x] rcone backup and associated configuration (see /etc/systemd/system/rclone-sync.service & /etc/systemd/system/rclone-sync.timer)
[x] UPS monitor and associated configuration (see /etc/nut directory)
[x] manual backup scripts (see /root/backup directory)
[ ] prometheus-node-exporter
[ ] prometheus-upsc-exporter (see /etc/systemd/system/prometheus-pve-exporter.service
[x] syslog exporting to promtail

billimek commented 3 years ago

Rough migration plan:

'handle' workloads above (probably backup configs for later reference in TrueNAS solution)
Stop nvr VM (uses NFS)
Scale-down the following k8s workloads that use NFS: flood, jackett, lazylibrarian, nzbget, plex, radarr, sonarr, tesla-dashcam, thanos, loki, minio
export tank zpool: zpool export tank
shutdown proxmox node
boot TrueNAS USB/ISO
Install TrueNAS to SSD Samsung_SSD_850_EVO_250GB_S21NNXBG304131Y (use same host IP of 10.0.7.7)
Import tank zpool and properly configure
Configure TrueNAS for NFS clients as before
Verify proxmox NFS client connectivity
Verify NFS client connectivity
Scale-up and verify k8s workloads NFS client connectivity
Go/No-Go decision on moving forward with TrueNAS
Remove proxmox node from proxmox cluster pvecm delnode proxmox from one of the other proxmox nodes

billimek commented 3 years ago

During step 4, proxmox would not cleanly export the tank zpool and I had to just forcefully shut it down. The subsequent step #8 still worked properly however
Step 9 had a snag in that TrueNAS mounts the zpool inside /mnt (e.g. /mnt/tank) while proxmox had them mounted at root. I did not want to mess with forcing trueNAS to mount it to a non-standard mountpoint and instead modified the proxmox and other NFS client configurations to mount them at the new mount-point.
Step 10 required fixing of permissions in /mnt/tank/proxmox/images to work properly
Step 12 required deleting and re-creating the nfs PVs and PVCs because the NFS 'path' is immutable

billimek commented 3 years ago

UPS configuration required manual setup outside of what the GUI can provide. If the GUI configuration is attempted again, it will reset the configuration. This is not ideal.

For future ref:

truenas# cat ups.conf
maxretry = 3

[ups]
        driver = usbhid-ups
        port = auto
        desc = "Cyber Power System, Inc. CP1500 AVR UPS"
        pollinterval = 20

truenas# cat upsd.conf
LISTEN 127.0.0.1 3493
LISTEN ::1 3493
MAXAGE 25

truenas# cat upsmon.conf
MONITOR ups@localhost:3493 1 upsmon <REDACTED>MASTER
#NOTIFYCMD "/usr/local/sbin/upssched"
NOTIFYCMD "/etc/local/nut/upscommand.sh"
NOTIFYFLAG ONBATT SYSLOG+EXEC
NOTIFYFLAG LOWBATT SYSLOG+EXEC
NOTIFYFLAG ONLINE SYSLOG+EXEC
NOTIFYFLAG COMMBAD SYSLOG+EXEC
NOTIFYFLAG COMMOK SYSLOG+EXEC
NOTIFYFLAG REPLBATT SYSLOG+EXEC
NOTIFYFLAG NOCOMM SYSLOG+EXEC
NOTIFYFLAG FSD SYSLOG+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+EXEC
SHUTDOWNCMD "/sbin/shutdown -p now"
POWERDOWNFLAG /etc/nokillpower
HOSTSYNC 15
NOCOMMWARNTIME 300
MINSUPPLIES 1
POLLFREQ 5
POLLFREQALERT 5
DEADTIME 25
RBWARNTIME 43200
FINALDELAY 5

billimek commented 3 years ago

It is not possible to install the starship prompt because there are no FreeBSD binaries readily available that work in TueNAS

billimek commented 3 years ago

rclone backup solution implemented using built-in 'cloud sync' features to back-up /mnt/tank/backups/borg/ and /mnt/tank/data/minio/velero/restic/ directories.

billimek commented 3 years ago

borgbackup solution implemented by creating a 'jail' with configuration and settings persisted in /mnt/tank/iocage/jails/borg. This jail is mounting /mnt/tank/backups/borg/ to /mnt/borg inside it's 'container'.

Added the necessary ssh keys and installed borg to the jail, and updated the client configurations (home, proxmox-b, proxmox-c).

billimek / homelab-infrastructure

convert 'proxmox' node from proxmox to truenas #21