[Research] RaspiBlitz on ZFS x86_64

[openoms]

This is worth a deeper look for redundant data storage (vs BTRFS): https://www.nasbeery.de/

The script is nice and simple (except pinging Google for time lol):

#!/bin/sh
# Set current Time
sudo date -s "$(curl -s --head http://google.com.au | grep ^Date: | sed 's/Date: //g')"

# Stop unattended Upgrades and wait to continue
sudo /etc/init.d/unattended-upgrades stop
i=0 
tput sc 
while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1 ; do
     case $(($i % 4)) in
         0 ) j="-" ;;
         1 ) j="\\" ;;
         2 ) j="|" ;;
         3 ) j="/" ;;
     esac
     tput rc
     echo -en "\r[$j] Waiting for unattended Upgrade to finish..." 
     sleep 0.5
     ((i=i+1)) 
done

# Install necessary Packages and ZFS
sudo apt update
sudo apt install -y samba zfs-dkms cockpit dialog 

# Start ZFS Module before reboot
sudo /sbin/modprobe zfs 

# Set first Samba Password
sudo smbpasswd -x ubuntu
(echo NasBeery2020; echo NasBeery2020) |sudo smbpasswd -a ubuntu

# Get ZFS Addon for Cockpit
git clone https://github.com/optimans/cockpit-zfs-manager.git
sudo cp -r cockpit-zfs-manager/zfs /usr/share/cockpit

# Install zfs-auto-snapshot and change Retention from 24 to 48h and 12 to 3 Month for more sense of usage
sudo apt install -y zfs-auto-snapshot
sudo sed -i 's/24/48/g' /etc/cron.hourly/zfs-auto-snapshot
sudo sed -i 's/12/3/g' /etc/cron.monthly/zfs-auto-snapshot

# change hostname
sudo sed -i 's/ubuntu/nasbeery/g' /etc/hostname

# create Mirror and force Deletion of existing Data
sudo zpool create -f -o autoexpand=on -o ashift=12 tank mirror sda sdb  

# create Share with Compression, Samba share has to be in smb.conf to work with Snapshots later
sudo zfs create -o compression=lz4 tank/share
sudo chmod -R 770 /tank
sudo chown -R ubuntu:root /tank

# check Mirror to be online, otherwise Power Indicator like LED or Buzzer
echo "PATH="/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"\n*/1 * * * * root echo 14 > /sys/class/gpio/export 2> /dev/null;echo out > /sys/class/gpio/gpio14/direction ; zpool import -fa -d /dev/ > /dev/null; zpool list| grep -q ONLINE; echo \$? > /sys/class/gpio/gpio14/value" | sudo tee  "/etc/cron.d/raidled"

# Add to smb.conf how ZFS Snapshots

echo "[share]\ncomment = Main Share\npath = /tank/share\nread only = No\nvfs objects = shadow_copy2\nshadow: snapdir = .zfs/snapshot\nshadow: sort = desc\nshadow: format = -%Y-%m-%d-%H%M\nshadow: snapprefix = ^zfs-auto-snap_\(frequent\)\{0,1\}\(hourly\)\{0,1\}\(daily\)\{0,1\}\(monthly\)\{0,1\}\nshadow: delimiter = -20\n" | sudo tee -a "/etc/samba/smb.conf"

sudo reboot

And the cockpit interface looks good and widely used: https://github.com/optimans/cockpit-zfs-manager

[openoms]

Front running the argument over ZFS must have ECC RAM as apparently ZFS helps with the diskIO errors and not worse with non-ECC RAM than EXT4 or other filesystems: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

The big difference is ZFS has no data-recovery tool available if the zpool gets corrupted (doesn't mount). This is a worse situation than on EXT4 if we assume that the data corrupted is not the channel.db (a non-redundant file).

If the only thing corrupted is a the valuable file, then it is the same bad situation on EXT4. The valuable file is also likely to be one of the first or only thing corrupted in any kind of failure (disk or RAM) since it is the most frequently written one. Corruption due to a disk error on ZFS(/BTRFS) is much less likely on ZFS/BTRFS due to the self-healing (checksumming) in the RAID.

ZFS/(BTRFS) is only a disadvantage in the rare case if the RAM causes the data corruption of a non-valuable file (far less common than disk-errors). The valuable file would remain recoverable on EXT4, but possibly not on ZFS.

The file corruptions due to bitflips in the RAM is what can be prevented by ECC RAM.

AFAIK ECC RAM is not available in any SBC-s yet, but always looking out for this. Building on server architecture is reasonably affordable with used hardware, but outside of the scope of RaspiBlitz (for now).

[openoms]

Now looking into if it would make sense to start with ZFS even for single disk setups, will comment/edit here.

[rootzoll]

As discussed on hackday ... if we switch RaspiBlitz to ZFS we might become one of the biggest projects that uses ZFS on ARM. This could be a very rocky ride with tapping into some early adopter problems.

[openoms]

Agreed, not for the RaspberryPi!

On the other hand I am planning to start a node on a Xeon based server with ECC RAM and will base it on ZFS.

Will try to deploy with the raspiblitz script, so would be useful to leave this open for further documentation of that.

[openoms]

I have Debian VM running the RaspiBlitz scripts on top of TrueNAS (FreeBSD - 6 disks in RAIDZ2). More details soon.

[rootzoll]

@openoms interesting :) .. keep us updated

[openoms]

Just to note the licence issues when installing ZFS to Debian: (would be only a problem is distributing ZFS with the RaspiBlitz image which is not planned currently)

[openoms]

The Debian VM did not survive long. Looked like a disk corruption, but had issues with failing low quality SSD-s and multiple resilvering too. Jails remained stable, but the VM corrupted.

In the forums Ubuntu is reported to have better compatibility so proceeded with another VM built on that. https://www.truenas.com/docs/scale/virtualization/

Will share my notes here (and update as needed):

# download image (to a pool in the TrueNAs console or upload in the WebUI from the desktop)
wget https://releases.ubuntu.com/20.04.3/ubuntu-20.04.3-desktop-amd64.iso

# create VM (with a 32 GB disk and a chosen USER)
# https://www.ithands-on.com/2020/11/setup-vm-on-truenas-example-with-ubuntu.html

# install Ubuntu with defaults

# in VNC or connected monitor
# login and open the terminal

# install the ssh server
sudo apt-get install openssh-server

# check own IP_ADDRESS
hostname -I

# to have the Raspiblitz Status scrern show switch to the `pi` user:
sudo su - pi

# connect with ssh from a Linux desktop:
ssh USER@IP_ADDRESS

# Build RaspiBlitz:
# download the build script 
wget https://raw.githubusercontent.com/rootzoll/raspiblitz/dev/build_sdcard.sh
# run
sudo bash build_sdcard.sh false false rootzoll dev

# switch off
sudo shutdown now

# mount a 1TB disk volume (had success recovering from a used virtual disk too with the blocks, app-data and raspiblit.conf present)
# in TrueNAS DEVICES set it to VirtIO for better performance

# start the VM

# login with the user: admin # password is: raspiblitz
# The IP address might change at this point.
# Have a look at the router or run `hostname -I` in VNC or on the connected display
ssh admin@IP_ADDRESS

# continue with the RaspiBlitz menu

Drive configuration inside the VM:

# without a disk mounted:
admin@HOSTNAME:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 55.4M  1 loop /snap/core18/2128
loop1    7:1    0   51M  1 loop /snap/snap-store/547
loop2    7:2    0 32.3M  1 loop /snap/snapd/12883
loop3    7:3    0 32.3M  1 loop /snap/snapd/12704
loop4    7:4    0 65.1M  1 loop /snap/gtk-common-themes/1515
loop5    7:5    0  219M  1 loop /snap/gnome-3-34-1804/72
vda    252:0    0   32G  0 disk 
├─vda1 252:1    0  512M  0 part /boot/efi
└─vda2 252:2    0 31.5G  0 part /

# Disk attached:
admin@HOSTNAME:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 55.4M  1 loop /snap/core18/2128
loop1    7:1    0  219M  1 loop /snap/gnome-3-34-1804/72
loop2    7:2    0 65.1M  1 loop /snap/gtk-common-themes/1515
loop3    7:3    0 32.3M  1 loop /snap/snapd/12704
loop4    7:4    0 32.3M  1 loop /snap/snapd/12883
loop5    7:5    0   51M  1 loop /snap/snap-store/547
vda    252:0    0   32G  0 disk 
├─vda1 252:1    0  512M  0 part /boot/efi
└─vda2 252:2    0 31.5G  0 part /
vdb    252:16   0    1T  0 disk 
└─vdb1 252:17   0 1024G  0 part /mnt/hdd

[openoms]

Ran into some serious speed issues and these settings helped with the Ubuntu VM on TrueNAS: https://www.brianessig.com/2021/08/slow-virtual-machine-speeds-on-truenas-solved/

[mrsieb]

The Debian VM did not survive long. Looked like a disk corruption, but had issues with failing low quality SSD-s and multiple resilvering too. Jails remained stable, but the VM corrupted.

In the forums Ubuntu is reported to have better compatibility so proceeded with another VM built on that. https://www.truenas.com/docs/scale/virtualization/

Will share my notes here (and update as needed):

# download image (to a pool in the TrueNAs console or upload in the WebUI from the desktop)
wget https://releases.ubuntu.com/20.04.3/ubuntu-20.04.3-desktop-amd64.iso

# create VM (with a 32 GB disk and a chosen USER)
# https://www.ithands-on.com/2020/11/setup-vm-on-truenas-example-with-ubuntu.html

# install Ubuntu with defaults

Well this works well with the actual master too, playing around with this since days. Not to run it in a VM environment but to test en redo it on a native ubuntu/debian laptop/Desktop installation.

The only "tricky" part is the mounting of the data disk and the install script is detecting the correct partition. I do this all with complete "Fresh" install. The only issue i got is TOR. Its enabled by default but seams not to be running or takes ages until you got peers for syncing. The log just response with:

connect() to 127.0.0.1:9050 failed after wait: Connection refused (111)
connect() to 127.0.0.1:9050 failed after wait: Connection refused (111)

If I disable TOR via the menu and reboot immediately 10 Peers are found and sync begins.

If i check the tor status after a fresh install i got:

admin@duck-VirtualBox:~/config.scripts$ bash internet.tor.sh status activated=1 torRunning=0 torFunctional=0 config='/etc/tor/torrc'

[openoms]

Thanks for testing @mrsieb! The partition detection should be improved in the current dev and now v1.7 branch also. It should detect the biggest (non-operating system) disk attached and not need manual intervention any more.

See the quick manual solution for the Tor problem in Ubuntu in https://github.com/rootzoll/raspiblitz/issues/2531

[mrsieb]

Thanks for testing @mrsieb! The partition detection should be improved in the current dev and now v1.7 branch also. It should detect the biggest (non-operating system) disk attached and not need manual intervention any more.

Yes this i see in the script that the biggest partition get used, BUT this only works if the disk has already a partition. Disk without that get detected as possible RAID candidate. And the Fresh install kills the System Disk.

Thx for the Tor Ubunut hint i will check this out.

[mrsieb]

6th optional parameter: TWEAK-BOOTDRIVE

Whats the advantage or disadvantage of this tweak? When to use and when not ?

Are there other tweaks witch should considered if the system runs on a x86_amd64 system? Like remove db tweaks in bitcoin.conf for raspi ?

[mrsieb]

On a pure debian install the baseSystem check is redundant and "debian" is not covered the script stops because of no TOR sources. Mentioned here https://github.com/rootzoll/raspiblitz/issues/2611

[mrsieb]

SETUP INFO
isSSD=1
hddCandidate='sda'
hddBytes=34359738368
hddGigaBytes=32
hddPartitionCandidate='sda1'
hddDataPartitionBytes=536870912
hddPartitionGigaBytes=0
hddFormat='vfat'
hddRaspiData=0
hddBlocksBitcoin=0
hddBlocksLitecoin=0
hddGotBlockchain=0
hddAdapterUSB=''
hddAdapterUSAP=0

RAID
isRaid=0
raidCandidate[0]='sdb 1000 GB ATA'
raidCandidates=1

admin@duck-VirtualBox:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   32G  0 disk
├─sda1   8:1    0  512M  0 part
├─sda2   8:2    0  513M  0 part /boot/efi
├─sda3   8:3    0    1K  0 part
└─sda5   8:5    0   31G  0 part /
sdb      8:16   0 1000G  0 disk
sr0     11:0    1 1024M  0 rom

This is how it looks if the new disk have no partion -> System disk get wiped if i countine with a FRESH SETUP. After creating a Partition , it gets dedected as data disk BUT still needs an reboot before to get realy used. I did not found the script to reinit this without.

SETUP INFO
isSSD=1
hddCandidate='sdb'
hddBytes=1073741824000
hddGigaBytes=1000
hddPartitionCandidate='sdb1'
hddDataPartitionBytes=1073740758528
hddPartitionGigaBytes=999
hddFormat=''
hddRaspiData=0
hddBlocksBitcoin=0
hddBlocksLitecoin=0
hddGotBlockchain=0
hddAdapterUSB=''
hddAdapterUSAP=0

RAID
isRaid=0
raidCandidates=0

admin@duck-VirtualBox:~$ lsblk

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   32G  0 disk
├─sda1   8:1    0  512M  0 part
├─sda2   8:2    0  513M  0 part /boot/efi
├─sda3   8:3    0    1K  0 part
└─sda5   8:5    0   31G  0 part /
sdb      8:16   0 1000G  0 disk
└─sdb1   8:17   0 1000G  0 part
sr0     11:0    1 1024M  0 rom

Anyhow, I have no idea what i have done differently last time. I tried it now with the dev and 1.7 repo (same behavior). The Setup exits on error state 14 Something does not realy work when creating a new lnd wallet. Passwords get asked but no seed is shown. And no macarons are generated, and end in inconsitent install state. Looks like something is executed in the wrong order. If i just choose Setup WITHOUT LND the setup gets done.

Anyhow Is there a way to manually remove some install flags, so i do not end in a "Reflashstate". Install step nr get stored in raspiblitz.info. Its some how hard to go back if anything goes wrong during the provisioning.

_provision.setup.sh
Prepare bitcoin 
mkdir: cannot create directory ‘/mnt/hdd/bitcoin’: File exists
File Exists: /mnt/hdd/bitcoin/bitcoin.conf --> 1
setting PASSWORD B
 OK -> RPC Password B changed
Reboot is needed
Optimizing RAM for Sync 
kbSizeRAM(3930024)
Detected RAM >=4GB --> optimizing bitcoin.conf
Start bitcoin (SETUP) 
This can take a while ..
(0/50) checking if bitcoind is running ...
 (1/50) checking if bitcoind is running ...
OK bitcoin startup successful
Prepare Lightning (lnd)
Setup LND
Creating new LND config ...
Init LND Service & Start
Starting LND Service ...
WALLET --> NEW
Copy LND Macaroons to user admin 
FAIL: setup: lnd no macaroons
ERROR - please report to development team
state(error) message(_provision.setup.sh fail)
https://github.com/rootzoll/raspiblitz#support
command to shutdown --> off

[mrsieb]

Thanks for testing @mrsieb! The partition detection should be improved in the current dev and now v1.7 branch also. It should detect the biggest (non-operating system) disk attached and not need manual intervention any more.

I did test now also with a Notebook and Ubunutu. ( sdb = 120 GB Flash) sda=2TBSSD every time when the installation try to delete the Harddisk, a "lost HDD" is detected end a reboot is triggered. This always happens on the mkfs command. I tried to play around in the blitz.datadrive.sh and change sleep times around this commands its feels like there is some timing problem on a native Nootebook compared to rasperypi.

[mrsieb]

..... "lost HDD" is detected end a reboot is triggered. This always happens on the mkfs command. I tried to play around in the blitz.datadrive.sh and change sleep times around this commands its feels like there is some timing problem on a native Nootebook compared to rasperypi.

I modified the sleep value in the loop to 30 https://github.com/rootzoll/raspiblitz/blob/4cf02caa43f6780139c4c924a9783d087559c47f/home.admin/_bootstrap.sh#L533 and the setup pass through...

[mrsieb]

Something does not realy work when creating a new lnd wallet. Passwords get asked but no seed is shown. And no macarons are generated, and end in inconsitent install state. Looks like something is executed in the wrong order. If i just choose Setup WITHOUT LND the setup gets done.

Choose C-Lightning is also working with a fresh wallet install

[openoms]

@mrsieb please move the experiences with the Ubuntu base on a laptop to a new issue. This is about ZFS.

[openoms]

The main approaches to use ZFS on X86 would be:

• Run Raspiblitz in a VM with the images kept on a ZFS volume. The underlying OS can be anything, including TrueNAS as above

• Use Debian or Ubuntu as the base OS, create a ZFS volume and mount it as /mnt/hdd

Using Linux as a base with ZFS would require some additional software to manage snasphots, scrubbing, notifications etc.

Sanoid is a policy-driven snapshot management tool for ZFS filesystems. https://github.com/jimsalterjrs/sanoid/ Includes syncoid to manage to off-site backups.

Would be great to find some well packaded ZFS manager for Linux. Unfortunately the earlier mention cockpit GUI is not maintained any more: https://github.com/optimans/cockpit-zfs-manager

End-of-Life

This software has reached its end-of-life and is no longer maintained.

TrueNAS Scale is a Debian based project, but it beta phase (not recommended for production yet): https://www.truenas.com/truenas-scale/ Will keep an eye on it as a replacement for the FreeBSD based TrueNAS. Even conversion might become possible.

Using either approach the biggest problem seems to be automating which disk / volume / partition to choose to become /mnt/hdd. For X86 we could avoid this by offering a manual step (with some dialog help) to choose from the available disks and make sure no boot or OS partition is getting overwritten.

[DamageIncorporated]

I am in the process of following @openoms notes to install raspiblitz on a ubuntu 22.04 VM on truenas 13.. Thank you for that writeup, as an amateur in these things I need that kind of help ;)

There were a few things I had to change, such as the start parameters of the script (see below). I also mounted /mnt/hdd before starting the script...

The Debian VM did not survive long. Looked like a disk corruption, but had issues with failing low quality SSD-s and multiple resilvering too. Jails remained stable, but the VM corrupted.

In the forums Ubuntu is reported to have better compatibility so proceeded with another VM built on that. https://www.truenas.com/docs/scale/virtualization/

Will share my notes here (and update as needed):

I hope my problem belongs here in the comments. Alternatively I could also open a new issue.

The build_sd_card script runs fine, i ran it with the options -w off (no wireless region: networking is handled by freenas) and -d headless (this is no raspberry pi, so no lcd and no hdmi-port).

The script stops with a "file not found" error, when trying to install "blitz.i2pd". When I encountered this, I disabled that line in section I2P by comment and restarted the whole script... The rest seems to run okay.. I hope this won't bite me in the future :)

bitcoin-23.0/share/man/man1/bitcoin-wallet.1
bitcoin-23.0/share/man/man1/bitcoind.1
alias bitcoinlog="sudo tail -n 30 -f /mnt/hdd/bitcoin/debug.log"
- Bitcoin install OK

build_sdcard.sh: Zeile 764: /home/admin/config.scripts/blitz.i2pd.sh: Datei oder Verzeichnis nicht gefunden
user@raspiblitz:~$

[DamageIncorporated]

Next Problem: In Step "Install core lightning" script /home/admin/config_scripts/cl.install.sh:

Sorry for the error messages translated to german, I hope these can help anyway:

Klone nach 'lightning' …
remote: Enumerating objects: 96507, done.
remote: Counting objects: 100% (287/287), done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 96507 (delta 116), reused 151 (delta 85), pack-reused 96220
Empfange Objekte: 100% (96507/96507), 80.60 MiB | 4.19 MiB/s, fertig.
Löse Unterschiede auf: 100% (74082/74082), fertig.

- Reset to version v0.11.2
HEAD ist jetzt bei c9587ab17 connectd: don't start connecting in parallel in peer_conn_closed.
Leider darf der Benutzer root »/home/admin/config.scripts/blitz.git-verify.sh rustyrussel https://raw.githubusercontent.com/ElementsProject/lightning/master/contrib/keys/rustyrussell.txt D9200E6CD1ADB8F1 v0.11.2« als bitcoin auf raspiblitz nicht ausführen.
mkamp@raspiblitz:~$ 

Can I continue from here or doesn't it make sense and I should rather give up on using a VM for RaspiBlitz? I realize this is not the focus of the project, but want to avoid plugging another Raspberry Pi when there's already a server running that has free resources.

[DamageIncorporated]

Last Status update: After just running the build script again, there was no more error message and now my node is syncing the blockchain. This is better than in the last four attempts, so I am happy for now :-)

[openoms]

This became offtopic and mostly outdated. Will close and continue in https://github.com/rootzoll/raspiblitz/issues/3590

[Silarous]

Just had a successful RaspiBlitz install in a Proxmox VM. Install went very smooth and had no issues. I'm running Proxmox VE 8.1.3 and used Debian 12.2.0 for the VM. Hardware is a Ryzen 5700g, 32GB DDR4 RAM, and a Samsung 990Pro 4TB NVME SSD.

Steps taken

1. Setup the VM in Proxmox to install Debian, CPU set to Host, and allotted a 64GB drive on the 990Pro to emulate the SD card.
2. Started the VM and installed Debian with no GUI.
3. Booted into the VM, switched to root and installed sudo with ' apt install sudo '
4. Downloaded the RaspiBlitz build script with ' wget https://raw.githubusercontent.com/rootzoll/raspiblitz/dev/build_sdcard.sh '
5. Ran the build script with ' sudo bash build_sdcard.sh -f true -b dev -d headless -t false -w off '
6. When the script finished the install I shut down the VM.
7. Configured a second 1.5TB SSD on the 990Pro for the VM to emulate the data drive and restarted the VM.
8. SSH into the VM and the setup took off just like on the Raspberry Pi 4.

So far I have Electrs, Mempool, JoininBox, and C-Lightning running successfully. I have Zeus connected to the node as well. Happy to say this install was completely painless and it runs MUCH faster than on the Pi. Just thought I'd share my experience.