Systemd service fails to start at boot - Failed to set up mount namespacing

[Shished]

This may be related to the issue #196 but I’m not sure. The beesd service is enabled but if fails to start during system startup. Here is journalctl output:

Oct 21 17:08:50 linux systemd[1]: Started Bees (d252ef93-c68c-48a8-bc79-5a59b28ed648).
Oct 21 17:08:50 linux systemd[635]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Failed to set up mount namespacing: /run/systemd/unit-root/disk/yuri/.beeshome-root: No such file or directory
Oct 21 17:08:50 linux systemd[635]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Failed at step NAMESPACE spawning /usr/sbin/beesd: No such file or directory
Oct 21 17:08:50 linux systemd[1]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Main process exited, code=exited, status=226/NAMESPACE
Oct 21 17:08:50 linux systemd[1]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Failed with result 'exit-code'.

Works fine after a manual service restart on a booted system.

Used config:

UUID=d252ef93-c68c-48a8-bc79-5a59b28ed648
WORK_DIR=/run/bees/
MNT_DIR="$WORK_DIR/mnt/$UUID"
BEESHOME="/disk/yuri/.beeshome-root"
BEESSTATUS="$WORK_DIR/$UUID.status"
OPTIONS="--strip-paths --no-timestamps --thread-count 12"
DB_SIZE=$((1024*1024*1024)) # 1G in bytes`

The target partition is on SSD and mounted as / and /home, /disk is mounted to a btrfs partition on HDD.

[kakra]

Do you have a split-usr setup? Do you mount btrfs only in late boot stage?

[Shished]

No, they are mounted normally, from fstab.

[kakra]

Could you show your fstab?

[Shished]

I'm using partuuids instead of uuids in fstab. Root and home uses uuid=d252ef93-c68c-48a8-bc79-5a59b28ed648 partition

PARTUUID=1e4353c1-a72f-4eef-bf02-3938159dad27   /           btrfs       rw,noatime,compress-force=zstd:4,ssd,space_cache=v2,subvol=/arch            0 0

PARTUUID=1e4353c1-a72f-4eef-bf02-3938159dad27   /home           btrfs       rw,noatime,compress-force=zstd:4,ssd,space_cache=v2,subvol=/home            0 0

# /dev/nvme0n1p1 UUID=D0D3-2D85 LABEL=EFI
PARTUUID=c5d4678c-01fd-4459-a6a6-7cd1f6b7711b   /boot       vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro   0 2

PARTUUID=83fddb96-def7-4ad3-b9e0-b595da5c2e6f   /disk       btrfs       rw,noatime,nofail,compress-force=zstd:6,space_cache=v2              0 0

PARTUUID=25a387c4-601b-492e-9a64-a5afdff39e92   swap        swap        defaults                                        0 0

[kakra]

Try adding the option x-systemd.automount to the /disk partition. Also, you can remove space_cache=v2 - that is a one-time mount option and will be stored in the superblock, it cannot be changed easily anyways (except when explicitly clearing the space cache).

The automount option mounts an automounter on the mount point at boot, and will actually mount it on first access, temporarily blocking processes accessing the device. This may sort out service ordering problems. If it helps but you don't like the automounter, you can then try adding RequiresMountsFor=/disk in the unit section of the service file instead. But I'd try with the automount option first (may also speed up booting). The problem may be that the bees service depends on basic.target which only depends on tmp directory presence.

WRT systemd, it is actually preferred to setup all late-boot file systems with the automounter so service starts are properly serialized automatically. This would be /home and /disk in your case.

Personally, I'm also mounting /boot with the automounter and an idle option x-systemd.idle-timeout=120 so it will automatically unmount after use. This helps protecting it from damage because there's usually no dirty outstanding cache for that file system.

[Shished]

I added x-systemd.automount to the mount options in fstab for /disk (also removed space_cache=v2) and after reboot bees service has started and works without problems. Thank you for your help!

[kakra]

@Zygo we should probably document that users should add x-systemd.automount to their late-boot btrfs mount points - or otherwise bees may simply start too early (because the systemd service knows nothing about which filesystems are needed by bees). Adding that option adds a synchronization point between bees startup (and possible other services accessing those fs) and the file system itself. It's probably needed anyways because you really don't want to start adding your file systems to the unit section of various services possible accessing late-boot mounted fs. This is most likely an issue for systems where btrfs (that one bees is working to be exact) is not the rootfs.

OTOH, replacing basic.target by local-fs.target may solve that as it auto-generates dependencies for all mount points marked "auto" in fstab, and all local paths listed in dependent systemd units.

[kakra]

No, they are mounted normally, from fstab.

BTW, with systemd, that is late boot stage... systemd only mounts the minimum of needed filesystems in the early boot stage (to be exact, it requires /var/tmp and /var to be ready, and forces /tmp to mount). Right after that, bees can start. On a fast system, it may then start bees before the needed file system is present in the mount table.

[kakra]

I researched a little, I think we should go with

[Unit]
After=local-fs.target

[Install]
WantedBy=multi-user.target

Actually, basic.target should not pull dependencies but be used as a synchronisation point only - thus WantedBy was wrong. And we need to start after local-fs.target has been reached because bees needs potentially all mounted local fs ready.