Systemd service cannot be started - beeshome is read only

[Shished]

Systemd service fails to start because beesd thinks that beeshome is read only. When beesd is started dirctly from the terminal it works fine. Target partition is used for / and /home, beeshome subvolume is located on other HDD with btrfs.

Oct 18 08:55:15 linux systemd[1]: Started Bees (d252ef93-c68c-48a8-bc79-5a59b28ed648). Oct 18 08:55:15 linux beesd[22153]: INFO: Find d252ef93-c68c-48a8-bc79-5a59b28ed648 in /etc/bees//d252ef93-c68c-48a8-bc79-5a59b28ed648.conf, use as conf Oct 18 08:55:15 linux beesd[22153]: INFO: Check: Disk exists Oct 18 08:55:15 linux beesd[22153]: INFO: Check: Disk with btrfs Oct 18 08:55:15 linux beesd[22153]: INFO: WORK DIR: /run/bees/ Oct 18 08:55:15 linux beesd[22153]: INFO: MOUNT DIR: /run/bees//mnt/d252ef93-c68c-48a8-bc79-5a59b28ed648 Oct 18 08:55:15 linux beesd[22168]: touch: невозможно выполнить touch для '/disk/yuri/.beeshome-root/beeshash.dat': Файловая система доступна только для чтения Oct 18 08:55:15 linux beesd[22173]: chmod: изменение прав доступа для '/disk/yuri/.beeshome-root/beeshash.dat': Файловая система доступна только для чтения Oct 18 08:55:15 linux beesd[22153]: bees version 0.7.r2.g84adbae Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: Masking signals Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: context constructed Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: Parsing option 'T' Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: Parsing option 'P' Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: Parsing option 'T' Oct 18 08:55:15 linux beesd[22153]: 2021-10-18 08:55:15 22153.22153<7> bees: Parsing option 'c' Oct 18 08:55:15 linux beesd[22153]: bees[22153]: using relative path /run/bees/mnt/d252ef93-c68c-48a8-bc79-5a59b28ed648/ Oct 18 08:55:15 linux beesd[22153]: bees[22153]: setting rlimit NOFILE to 10340 Oct 18 08:55:15 linux beesd[22153]: bees[22153]: setting worker thread pool maximum size to 12 Oct 18 08:55:15 linux beesd[22153]: bees[22153]: setting root path to '/run/bees/mnt/d252ef93-c68c-48a8-bc79-5a59b28ed648' Oct 18 08:55:15 linux beesd[22153]: bees[22153]: set_root_path /run/bees/mnt/d252ef93-c68c-48a8-bc79-5a59b28ed648 Oct 18 08:55:15 linux beesd[22153]: bees[22153]: set_root_fd /run/bees/mnt/d252ef93-c68c-48a8-bc79-5a59b28ed648 Oct 18 08:55:15 linux beesd[22153]: bees[22153]: BeesStringFile /disk/yuri/.beeshome-root/beescrawl.dat max size 16M Oct 18 08:55:15 linux beesd[22153]: bees[22153]: btrfs send workaround disabled Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Scan mode set to 0 (0) Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Starting bees main loop... Oct 18 08:55:15 linux beesd[22153]: bees[22153]: BeesThread exec progress_report Oct 18 08:55:15 linux beesd[22153]: bees[22153]: BeesThread exec status_report Oct 18 08:55:15 linux beesd[22153]: progress_report[22187]: Starting thread progress_report Oct 18 08:55:15 linux beesd[22153]: bees[22153]: BeesStringFile /disk/yuri/.beeshome-root/beesstats.txt max size 16M Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Creating new hash table 'beeshash.dat.tmp' Oct 18 08:55:15 linux beesd[22153]: status_report[22188]: Starting thread status_report Oct 18 08:55:15 linux beesd[22153]: status_report[22188]: Writing status to file '/run/bees//d252ef93-c68c-48a8-bc79-5a59b28ed648.status' every 1 sec Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Thread hash_prefetch no thread ptr Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Thread hash_writeback no thread ptr Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Oct 18 08:55:15 linux beesd[22153]: bees[22153]: Oct 18 08:55:15 linux beesd[22153]: bees[22153]: EXCEPTION Oct 18 08:55:15 linux beesd[22153]: bees[22153]: exception type std::system_error: openat: dir_fd 4 /disk/yuri/.beeshome-root name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system Oct 18 08:55:15 linux beesd[22153]: bees[22153]: *** Oct 18 08:55:15 linux beesd[22153]: terminate called after throwing an instance of 'std::system_error' Oct 18 08:55:15 linux beesd[22153]: what(): openat: dir_fd 4 /disk/yuri/.beeshome-root name 'beeshash.dat.tmp' mode 700 flags O_CLOEXEC|O_CREAT|O_EXCL|O_LARGEFILE|O_NOATIME|O_NOCTTY|O_NOFOLLOW|O_NONBLOCK|O_WRONLY at fd.cc:276: Read-only file system Oct 18 08:55:15 linux systemd[1]: Started Process Core Dump (PID 22189/UID 0). Oct 18 08:55:15 linux systemd-coredump[22190]: Process 22153 (bees) of user 0 dumped core.

                                           Found module linux-vdso.so.1 with build-id: 914dd32239383cdcd688253e6c7c9f194bb84a68
                                           Found module ld-linux-x86-64.so.2 with build-id: 040cc3dd10461562f177df39e3be2f3704258c3c
                                           Found module libc.so.6 with build-id: 4b406737057708c0e4c642345a703c47a61c73dc
                                           Found module libgcc_s.so.1 with build-id: 7f8508bb914546ada778809b64b99d234337d835
                                           Found module libm.so.6 with build-id: 2b8fd1f869ecab4e0b55e92f2f151897f6818acf
                                           Found module libstdc++.so.6 with build-id: 8ab0e57054dd1dcba681f217016afc6a4e639783
                                           Found module libpthread.so.0 with build-id: 07c8f95b4f3251d08550217ad8a1f31066229996
                                           Found module bees with build-id: 8183135d9155ce2d495c11493fc95bb893d89f06
                                           Stack trace of thread 22153:
                                           #0  0x00007f8fafd7fd22 raise (libc.so.6 + 0x3cd22)
                                           #1  0x00007f8fafd69862 abort (libc.so.6 + 0x26862)
                                           #2  0x00007f8fb0107802 _ZN9__gnu_cxx27__verbose_terminate_handlerEv (libstdc++.so.6 + 0x99802)
                                           #3  0x00007f8fb0113c8a _ZN10__cxxabiv111__terminateEPFvvE (libstdc++.so.6 + 0xa5c8a)
                                           #4  0x00007f8fb0113cf7 _ZSt9terminatev (libstdc++.so.6 + 0xa5cf7)
                                           #5  0x00007f8fb0113fe1 __cxa_rethrow (libstdc++.so.6 + 0xa5fe1)
                                           #6  0x000055ae641aeb83 n/a (bees + 0x1db83)
                                           #7  0x000055ae641b5a83 n/a (bees + 0x24a83)
                                           #8  0x00007f8fafd6ab25 __libc_start_main (libc.so.6 + 0x27b25)
                                           #9  0x000055ae641b62ee n/a (bees + 0x252ee)

                                           Stack trace of thread 22176:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22177:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22181:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22175:
                                           #0  0x00007f8fb0296600 __lll_lock_wait (libpthread.so.0 + 0x12600)
                                           #1  0x00007f8fb028f503 __pthread_mutex_lock (libpthread.so.0 + 0xb503)
                                           #2  0x000055ae64245d59 n/a (bees + 0xb4d59)
                                           #3  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #4  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #5  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22182:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22180:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22183:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22187:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293842 pthread_cond_clockwait (libpthread.so.0 + 0xf842)
                                           #2  0x000055ae641ca16f n/a (bees + 0x3916f)
                                           #3  0x000055ae64224b84 n/a (bees + 0x93b84)
                                           #4  0x000055ae64210bbc n/a (bees + 0x7fbbc)
                                           #5  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #6  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #7  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22184:
                                           #0  0x00007f8fafe3c3fb __mprotect (libc.so.6 + 0xf93fb)
                                           #1  0x00007f8fafdc9c0a new_heap (libc.so.6 + 0x86c0a)
                                           #2  0x00007f8fafdcc144 arena_get2.part.0 (libc.so.6 + 0x89144)
                                           #3  0x00007f8fafdce009 tcache_init.part.0 (libc.so.6 + 0x8b009)
                                           #4  0x00007f8fafdcf3ee __libc_calloc (libc.so.6 + 0x8c3ee)
                                           #5  0x00007f8fafd82c73 __cxa_thread_atexit_impl (libc.so.6 + 0x3fc73)
                                           #6  0x00007f8fb01121be __cxa_thread_atexit (libstdc++.so.6 + 0xa41be)
                                           #7  0x000055ae64246540 n/a (bees + 0xb5540)
                                           #8  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #9  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #10 0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22188:
                                           #0  0x00007f8fb0296600 __lll_lock_wait (libpthread.so.0 + 0x12600)
                                           #1  0x00007f8fb028f503 __pthread_mutex_lock (libpthread.so.0 + 0xb503)
                                           #2  0x000055ae6423f884 n/a (bees + 0xae884)
                                           #3  0x000055ae641d0261 n/a (bees + 0x3f261)
                                           #4  0x000055ae64224b84 n/a (bees + 0x93b84)
                                           #5  0x000055ae64210bbc n/a (bees + 0x7fbbc)
                                           #6  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #7  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #8  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22178:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22179:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22186:
                                           #0  0x00007f8fb02998ca __futex_abstimed_wait_common64 (libpthread.so.0 + 0x158ca)
                                           #1  0x00007f8fb0293270 pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0xf270)
                                           #2  0x00007f8fb013af01 __gthread_cond_wait (libstdc++.so.6 + 0xccf01)
                                           #3  0x000055ae642464cf n/a (bees + 0xb54cf)
                                           #4  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #5  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #6  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

                                           Stack trace of thread 22185:
                                           #0  0x00007f8fb0296600 __lll_lock_wait (libpthread.so.0 + 0x12600)
                                           #1  0x00007f8fb028f503 __pthread_mutex_lock (libpthread.so.0 + 0xb503)
                                           #2  0x000055ae64245d59 n/a (bees + 0xb4d59)
                                           #3  0x00007f8fb01413c4 execute_native_thread_routine (libstdc++.so.6 + 0xd33c4)
                                           #4  0x00007f8fb028d259 start_thread (libpthread.so.0 + 0x9259)
                                           #5  0x00007f8fafe415e3 __clone (libc.so.6 + 0xfe5e3)

Oct 18 08:55:15 linux systemd[1]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Main process exited, code=dumped, status=6/ABRT Oct 18 08:55:15 linux systemd[1]: beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service: Failed with result 'core-dump'. Oct 18 08:55:15 linux systemd[1]: systemd-coredump@9-22189-0.service: Deactivated successfully.

Used bees version 0.7.r2.g84adbae installed from bees-git AUR on Arch Linux.

[kakra]

Could you show your file /etc/bees//d252ef93-c68c-48a8-bc79-5a59b28ed648.conf?

[Shished]

UUID=d252ef93-c68c-48a8-bc79-5a59b28ed648
WORK_DIR=/run/bees/
MNT_DIR="$WORK_DIR/mnt/$UUID"
BEESHOME="/disk/yuri/.beeshome-root"
BEESSTATUS="$WORK_DIR/$UUID.status"
OPTIONS="--strip-paths --no-timestamps --thread-count 12"
DB_SIZE=$((1024*1024*1024)) # 1G in bytes`

[kakra]

Okay, so you've put .beeshome onto a different disk? In that case, you should edit your service instance to add another writable path:

systemctl edit beesd@d252ef93-c68c-48a8-bc79-5a59b28ed648.service

Insert this text:

[Service]
ReadWritePaths=/disk/yuri/.beeshome-root

@Zygo We should probably document this. I cannot currently think of a way to do this differently other than moving a configuration file to beeshome which configures which FS to work on (which could default to the FS that .beeshome is on), and then instantiating the systemd service from the beeshome path...

Maybe we should say that .beeshome on a different FS is an unsupported configuration.

[Shished]

Thanks, this edit fixed the problem. I put beeshome on other disk because docs says "The hash table trickle-writes to disk at 4GB/hour to beeshash.dat" . That partition is located on SSD and I'm worried about its health. Also, docs says that

There are no special requirements for bees hash table storage--.beeshome could be stored on a different btrfs filesystem, ext4, or even CIFS.

[kakra]

Yep, it's a totally valid option but cannot be supported automatically with the locked down service - at least not without mangling some of the configuration logic.

SSD is a valid concern. Personally, I'm using bcache-backed btrfs (which stores hot data on SSD while the rest remains on set of HDDs), including a kernel patch which forces bcache-bypass for idle IO priority. Since bees runs with idle IO prio, it will thus bypass the SSD for writes.

[Zygo]

4 GB per hour is 35TB per year, which takes decades to wear out a modern SSD [1] unless it's a very old or low-budget model. It should stop writing once the entire table has been flushed (though it doesn't take very much new data to force the whole table to be updated). I don't worry about the writes from the hash table on my SSDs--I worry a lot more about the btrfs metadata writes from dedupe, which can be much larger, especially with snapshots.

[1] 600 TB TBW at 200% utilization = 34 years

[kakra]

Well, mine says (running as a bcache cache dev):

Device Model:     Samsung SSD 860 EVO 1TB

  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       8937
177 Wear_Leveling_Count     0x0013   075   075   000    Pre-fail  Always       -       377

which would make it 8937 * 75 / (100 - 75) = 26811 hours = 3 years left before smartctl would consider the drive worn out. Usually, the drives last much longer than announced. But still that's the prediction from smartctl and thus the manufacturer. I'd expect the drive to at least slow down after this threshold is reached. Or is this calculation completely bogus?

The power on hours itself calculate down to around 50 years of operation left, tho. Not sure which value is more trustworthy.

8937 matches the ages of the drive in hours, it's about one year old.

[Zygo]

The calculation tells you when the drive reaches the end of its TBW warranty (100% of endurance used, which is a complicated function of the lifetime of several different flash types on drives with SLC caches, and the size and number of writes).

When the drive hits 100% TBW, it could fail next month, next decade, or last summer. It is much like time-based warranties where the manufacturer guarantees only a few years, but a drive in good environmental conditions will often last many times longer.

A typical caching workload can generate far more than 4GB of writes per hour. I have machines running NVME caches that write 4GB every 3 minutes. They would run an 860 EVO to the end of its warranty in 8 months if the EVO didn't have crippling latency problems.

Another way to look at 4GB/hour is 0.1% of the drive's warranty per week.

[kakra]

They would run an 860 EVO to the end of its warranty in 8 months if the EVO didn't have crippling latency problems

OT: yep, I'm thinking about going back to Crucial again... I feel it had better latency although throughput was lower. But I probably prefer better latency. What drive would you recommend?

[Zygo]

Samsung PRO models do the job, but they are the most expensive consumer drives for what they do (you can pay more, but those drives are specialized for video production or industrial applications).

Seagate Ironwolf SSD has consistent latency, huge TBW, and no firmware issues that I know of, and it's explicitly designed to be hammered by a cache driver all day. Nytro has more speed, but the engineering changes to get the slightly lower write latency cost almost 4x the drive write endurance. Either one has the TBW of at least 3 860 EVO's but slower writes than PRO (they won't saturate a SATA3 port).

WD Red SSD doesn't have the EVO's huge latency spikes, but about the same average performance and TBW. It's cheaper than the Seagates so it might be a better fit for lighter cache workloads.

Sabrent Rocket NVME devices are cheaper and perform within ~10% of the comparable Samsung models, but I've had multiple bus reset issues with one Sabrent and I've only been using them for a few weeks. I'll try to find some non-critical job for the devices I have so far to do, and not buy any more of the current models.

I still have Crucial MX and BX devices in service. They're slower (BX maybe too slow), but they work.

Everything cheaper in my fleet is dead now. The good ones chose a day to become a brick, the bad ones corrupted data for a few months first.