Store persistent service data on a separate USB

[scottmuc]

The Raspberry PI has a 2TB exFAT formatted USB drive attached (and PI only supports up to USB 2.0).

ansible@raspberrypi:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       1.9T  809G  1.1T  44% /mnt/usb
...

ansible@raspberrypi:~ $ cat /etc/fstab
...
UUID=5A47-F8E2 /mnt/usb exfat defaults,auto,users,rw,nofail,umask=000 0 0

/mnt/usb contains the following directories:

code                  
grafana         <-- used by a service running on the PI
inbox
music           <-- used by a service running on the PI
navidrome       <-- used by a service running on the PI
old-home-dirs
photos
presentations
prometheus      <-- used by a service running on the PI
videos
youtube

The directories for grafana, navidrome, and prometheus store state and data that I want to persist across machine repaves. music is needed by navidrome, but this could be replaced by a network share from my Windows PC.

All of /mnt/usb is made available to my LAN with Samba. It's not offering any other benefit, so it's equivalent to a slow USB attached storage to my other devices.

Planned End State

• [x] PI's services use one of my spare 32GB drives to store logs/data/state
• [x] PI reads music from a share exposed on my Windows PC (cifs-fuse?)
• [x] Samba uninstalled
• [x] exFAT related drivers uninstalled
• [x] USB drive is attached to Windows PC (potentially plan a migration off of exFAT)
• [x] Repave instructions reflect new/deleted steps

All of these will have their own set of tasks to sort through.

[scottmuc]

The tail of dmesg displayed the following after I plugged in the drive:

[2346642.734783] usb 1-1.1.2: new high-speed USB device number 6 using dwc_otg
[2346642.836741] usb 1-1.1.2: New USB device found, idVendor=0781, idProduct=5581, bcdDevice= 1.00
[2346642.836783] usb 1-1.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2346642.836799] usb 1-1.1.2: Product: Ultra
[2346642.836811] usb 1-1.1.2: Manufacturer: SanDisk
[2346642.836822] usb 1-1.1.2: SerialNumber: 0501bb2085dfe237292096ef3c1c0de0b0111fae645e617e33ce0ad14c275
8fd88f90000000000000000000020704a6400171610815581072728877b
[2346642.837923] usb-storage 1-1.1.2:1.0: USB Mass Storage device detected
[2346642.838655] scsi host1: usb-storage 1-1.1.2:1.0
[2346643.852348] scsi 1:0:0:0: Direct-Access     SanDisk  Ultra            1.00 PQ: 0 ANSI: 6
[2346643.853560] sd 1:0:0:0: Attached scsi generic sg1 type 0
[2346643.859473] sd 1:0:0:0: [sdb] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB)
[2346643.860755] sd 1:0:0:0: [sdb] Write Protect is off
[2346643.860778] sd 1:0:0:0: [sdb] Mode Sense: 43 00 00 00
[2346643.861326] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[2346643.885942]  sdb: sdb1
[2346643.887301] sd 1:0:0:0: [sdb] Attached SCSI removable disk

The device is sufficient since the directories currently use:

prometheus - 366M grafana - 2.8M navidrome - 1.8G (album cover cache)

I'm going to format the drive with ext4, but use ansible to configure the mount point.

• [x] format the device
• [x] add ansible code to mount
• [x] stop grafana, navidrome, prometheus
• [x] mv directories over to new path
• [x] update configuration of each service to point to the new path
• [x] start all the services and verify they are working

[scottmuc]

Device formatted:

ansible@raspberrypi:~ $ sudo dd if=/dev/zero of=/dev/sdb status=progress bs=10M
30754734080 bytes (31 GB, 29 GiB) copied, 2406 s, 12.8 MB/s
dd: error writing '/dev/sdb': No space left on device
2935+0 records in
2934+0 records out
30765219840 bytes (31 GB, 29 GiB) copied, 2417 s, 12.7 MB/s
ansible@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda           8:0    0  1.8T  0 disk
└─sda1        8:1    0  1.8T  0 part /mnt/usb
sdb           8:16   1 28.7G  0 disk
mmcblk0     179:0    0 14.9G  0 disk
├─mmcblk0p1 179:1    0  512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 14.4G  0 part /
ansible@raspberrypi:~ $ sudo fdisk /dev/sdb

Welcome to fdisk (util-linux 2.38.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS (MBR) disklabel with disk identifier 0x7364d878.

Command (m for help): g
Created a new GPT disklabel (GUID: 6B52BAEE-4121-F042-A925-A85BDB8B9830).

Command (m for help): n
Partition number (1-128, default 1):
First sector (2048-60088286, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-60088286, default 60086271):

Created a new partition 1 of type 'Linux filesystem' and of size 28.7 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

ansible@raspberrypi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda           8:0    0  1.8T  0 disk
└─sda1        8:1    0  1.8T  0 part /mnt/usb
sdb           8:16   1 28.7G  0 disk
└─sdb1        8:17   1 28.7G  0 part
mmcblk0     179:0    0 14.9G  0 disk
├─mmcblk0p1 179:1    0  512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0 14.4G  0 part /
ansible@raspberrypi:~ $ sudo mkfs.ext4 -L vcapstore -m 1 -b 4096 /dev/sdb1
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 7510528 4k blocks and 1880480 inodes
Filesystem UUID: e15af423-34a5-445c-978b-c85288f3a90b
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

• this was a handy partitioning and formatting tutorial

[scottmuc]

Incoming ansible code successfully mounted the drive:

ansible@raspberrypi:~ $ df -h -T
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  318M     0  318M   0% /dev
tmpfs          tmpfs      91M   11M   81M  12% /run
/dev/mmcblk0p2 ext4       15G  3.3G   11G  25% /
tmpfs          tmpfs     454M     0  454M   0% /dev/shm
tmpfs          tmpfs     5.0M   12K  5.0M   1% /run/lock
/dev/mmcblk0p1 vfat      510M   63M  448M  13% /boot/firmware
/dev/sda1      exfat     1.9T  809G  1.1T  44% /mnt/usb
tmpfs          tmpfs      91M     0   91M   0% /run/user/1001
/dev/sdb1      ext4       29G   24K   28G   1% /mnt/vcapstore
ansible@raspberrypi:~ $ ls -la /mnt/vcapstore/
total 24
drwxr-xr-x 3 root root  4096 Apr 20 16:20 .
drwxr-xr-x 4 root root  4096 Apr 20 16:32 ..
drwx------ 2 root root 16384 Apr 20 16:20 lost+found

[scottmuc]

Needed to update the permissions to the copied files and everything works!

ansible@raspberrypi:/mnt/usb $ ls -la /mnt/vcapstore/
total 36
drwxr-xr-x  6 root       root   4096 Apr 20 16:42 .
drwxr-xr-x  4 root       root   4096 Apr 20 16:32 ..
drwxr-xr-x  7 grafana    admin  4096 Apr 20 17:11 grafana
drwx------  2 root       root  16384 Apr 20 16:20 lost+found
drwxr-xr-x  3 navidrome  admin  4096 Apr 20 17:12 navidrome
drwxr-xr-x 29 prometheus admin  4096 Apr 20 17:06 prometheus

Interestingly enough, with the new filesystem the files take only 543M now:

/dev/sdb1 29G 543M 28G 2% /mnt/vcapstore

[scottmuc]

The above commit seems to be working! Navidrome needs to re-index all the music so this will take a while. I do see that my Windows PC is averaging 20Mbps of outbound traffic so the share seems to be working.

I can't say with confidence what I did on Windows to make it work. This should be interesting when I repave my PC. This also means that I need to ensure my Windows PC doesn't sleep (or that I have the albums that I want to listen to cached).

This is a good time to pause and think about how I want to design a NAS. That'll be an issue for another time.

[scottmuc]

Manually deleted samba and exfat-fuse and all no-longer needed dependencies. Also removed the USB drive from /etc/fstab.

• [x] reboot PI after navidrome finishes indexing.

[scottmuc]

Everything survived the reboot! This went surprisingly well, but I think there's going to be pain with my share from the Windows PC (machine going to sleep and/or other stuff).