search

Zygo / bees

Best-Effort Extent-Same, a btrfs dedupe agent
GNU General Public License v3.0
630 stars 57 forks source link

cannot create directory ‘/run/bees/’: Read-only file system #195

Closed dunk1452 closed 2 years ago

dunk1452 commented 2 years ago

Hello, i compiled and installed bees yesterday on my Debian system running the Proxmox kernel, and ran into this issue when starting beesd.service:

❯ systemctl start beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service
❯ systemctl status beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service
● beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service - Bees (1d3b9432-fe63-4076-a3d8-444a4e987260)
     Loaded: loaded (/lib/systemd/system/beesd@.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2021-10-12 18:27:38 NZDT; 7s ago
       Docs: https://github.com/Zygo/bees
    Process: 14967 ExecStart=/usr/sbin/beesd --no-timestamps 1d3b9432-fe63-4076-a3d8-444a4e987260 (code=exited, status=1/FAILURE)
   Main PID: 14967 (code=exited, status=1/FAILURE)
        CPU: 21ms

Oct 12 18:27:38 pve systemd[1]: Started Bees (1d3b9432-fe63-4076-a3d8-444a4e987260).
Oct 12 18:27:38 pve beesd[14967]: INFO: Find 1d3b9432-fe63-4076-a3d8-444a4e987260 in /etc/bees//1d3b9432-fe63-4076-a3d8-444a4e987260.conf, us>
Oct 12 18:27:38 pve beesd[14967]: INFO: Check: Disk exists
Oct 12 18:27:38 pve beesd[14967]: INFO: Check: Disk with btrfs
Oct 12 18:27:38 pve beesd[14967]: INFO: WORK DIR: /run/bees/
Oct 12 18:27:38 pve beesd[14979]: mkdir: cannot create directory ‘/run/bees/’: Read-only file system
Oct 12 18:27:38 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Main process exited, code=exited, status=1/FAILURE
Oct 12 18:27:38 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Failed with result 'exit-code'.

Seems to be a problem with the current repo? Not sure because i still have an older version that seems to work fine.

ghost commented 2 years ago

+1, same issue

kakra commented 2 years ago

Could you try adding

[Service]
RuntimeDirectory=bees
WorkingDirectory=/run/bees

to the systemd service unit?

ghost commented 2 years ago

that seems to've fixed it from my end, thanks

dunk1452 commented 2 years ago

Well I tried that but now I'm getting a different error:

Oct 13 23:21:28 pve systemd[1]: Started Bees (1d3b9432-fe63-4076-a3d8-444a4e987260).
Oct 13 23:21:28 pve beesd[21731]: INFO: Find 1d3b9432-fe63-4076-a3d8-444a4e987260 in /etc/bees//1d3b9432-fe63-4076-a3d8-444a4e987260.conf, use as conf
Oct 13 23:21:28 pve beesd[21731]: INFO: Check: Disk exists
Oct 13 23:21:28 pve beesd[21731]: INFO: Check: Disk with btrfs
Oct 13 23:21:28 pve beesd[21731]: INFO: WORK DIR: /run/bees/
Oct 13 23:21:28 pve beesd[21731]: INFO: MOUNT DIR: /run/bees//mnt/1d3b9432-fe63-4076-a3d8-444a4e987260
Oct 13 23:21:28 pve beesd[21747]: touch: cannot touch '/mnt/nfs/ssd_btrfs/.btrfs_beeshome/beeshash.dat': Read-only file system
Oct 13 23:21:28 pve beesd[21752]: chmod: changing permissions of '/mnt/nfs/ssd_btrfs/.btrfs_beeshome/beeshash.dat': Read-only file system
Oct 13 23:21:28 pve beesd[21731]: bees version v0.7-1-g12e8065
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Masking signals
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: context constructed
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'T'
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'P'
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'c'
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'C'
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'g'
Oct 13 23:21:28 pve beesd[21731]: 2021-10-13 23:21:28 21731.21731<7> bees: Parsing option 'G'
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: using relative path /run/bees/mnt/1d3b9432-fe63-4076-a3d8-444a4e987260/
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: setting rlimit NOFILE to 10340
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: setting load average target to 4
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: setting worker thread pool minimum size to 2
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: setting worker thread pool maximum size to 3
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: setting root path to '/run/bees/mnt/1d3b9432-fe63-4076-a3d8-444a4e987260'
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: set_root_path /run/bees/mnt/1d3b9432-fe63-4076-a3d8-444a4e987260
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: set_root_fd /run/bees/mnt/1d3b9432-fe63-4076-a3d8-444a4e987260
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: BeesStringFile /mnt/nfs/ssd_btrfs/.btrfs_beeshome/beescrawl.dat max size 16M
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: btrfs send workaround disabled
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Scan mode set to 0 (0)
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Starting bees main loop...
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: BeesThread exec progress_report
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: BeesThread exec status_report
Oct 13 23:21:28 pve beesd[21731]: progress_report[21772]: Starting thread progress_report
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: BeesStringFile /mnt/nfs/ssd_btrfs/.btrfs_beeshome/beesstats.txt max size 16M
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Creating new hash table 'beeshash.dat.tmp'
Oct 13 23:21:28 pve beesd[21731]: status_report[21773]: Starting thread status_report
Oct 13 23:21:28 pve beesd[21731]: status_report[21773]: Writing status to file '/run/bees//1d3b9432-fe63-4076-a3d8-444a4e987260.status' every 1 sec
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Thread hash_prefetch no thread ptr
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Thread hash_writeback no thread ptr
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: *** EXCEPTION ***
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:  exception type std::system_error: openat: dir_fd 4 /mnt/nfs/ssd_btrfs/.btrfs_beeshome name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: ***
Oct 13 23:21:28 pve beesd[21731]: terminate called after throwing an instance of 'std::system_error'
Oct 13 23:21:28 pve beesd[21731]:   what():  openat: dir_fd 4 /mnt/nfs/ssd_btrfs/.btrfs_beeshome name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system
Oct 13 23:21:28 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Main process exited, code=killed, status=6/ABRT
Oct 13 23:21:28 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Failed with result 'signal'.

This is the systemctl output:

❯ systemctl status beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service | cat
● beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service - Bees (1d3b9432-fe63-4076-a3d8-444a4e987260)
     Loaded: loaded (/lib/systemd/system/beesd@.service; disabled; vendor preset: enabled)
     Active: failed (Result: signal) since Wed 2021-10-13 23:21:28 NZDT; 1min 5s ago
       Docs: https://github.com/Zygo/bees
    Process: 21731 ExecStart=/usr/sbin/beesd --no-timestamps 1d3b9432-fe63-4076-a3d8-444a4e987260 (code=killed, signal=ABRT)
   Main PID: 21731 (code=killed, signal=ABRT)
        CPU: 32ms

Oct 13 23:21:28 pve beesd[21731]: bees[21731]: Thread hash_writeback no thread ptr
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: *** EXCEPTION ***
Oct 13 23:21:28 pve beesd[21731]: bees[21731]:         exception type std::system_error: openat: dir_fd 4 /mnt/nfs/ssd_btrfs/.btrfs_beeshome name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system
Oct 13 23:21:28 pve beesd[21731]: bees[21731]: ***
Oct 13 23:21:28 pve beesd[21731]: terminate called after throwing an instance of 'std::system_error'
Oct 13 23:21:28 pve beesd[21731]:   what():  openat: dir_fd 4 /mnt/nfs/ssd_btrfs/.btrfs_beeshome name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system
Oct 13 23:21:28 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Main process exited, code=killed, status=6/ABRT
Oct 13 23:21:28 pve systemd[1]: beesd@1d3b9432-fe63-4076-a3d8-444a4e987260.service: Failed with result 'signal'.
kakra commented 2 years ago

that seems to've fixed it from my end, thanks

@TAK-YON We probably don't need the WorkingDirectory setting. Does it still work with just the RuntimeDirectory setting?

ghost commented 2 years ago

seems to still be working

should probably make a pull to update the script to have this by default

kakra commented 2 years ago

Yeah, that was an oversight by me when I reviewed the original pull request. I actually use that config line in my local deployment but hadn't had enough time to test the hardening-PR out. I wonder why it hadn't been a problem for the author, tho. Maybe they disabled state file writes?

The line in question actually pre-creates /run/bees and makes it writable to the service. Before the hardening, it was created by other means, and bees would have write-access to it anyways. I'm just using that in my local deployment because I do not use the wrapper script (which creates this directory) but run bees directly - and the easiest fix was letting systemd create that directory. Seems to take care of the write permissions, too, as we've seen now.

@Zygo This is worth a hotfix I believe as otherwise the service won't start under systemd. Do you want to just add a commit with this line added, or should I create a PR (for a one-line change)?

kakra commented 2 years ago

Well I tried that but now I'm getting a different error:

This should be reported separately then. You can use the three-dot menu in your comment to create a new issue from it.

Zygo commented 2 years ago

Is the different error due to a BEESHOME that isn't on the target filesystem? That might be a special case that we have to document.

@kakra is the one line "RuntimeDirectory=bees" in the Service section of scripts/beesd@.service.in? I can add that.

kakra commented 2 years ago

is the one line "RuntimeDirectory=bees" in the Service section of scripts/beesd@.service.in? I can add that.

Yep. Thanks.