search

aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Maybe disable CoW on Btrfs (and others) for Pack-files #110

Open dev-zero opened 3 years ago

dev-zero commented 3 years ago

Modern filesystems like Btrfs do a Copy-on-write. When adding data to a large file this may lead to fragmentation, similar to what can be seen in virtual machine images. The object store should therefore probably detect these filesystems (btrfs, zfs) and disable CoW on the pack files, see also this stackexchange for a discussion on CoW for VM images.

Note: I have not benchmarked this yet.

dev-zero commented 3 years ago

For Btrfs (and also zfs) this is done with chattr +C file on the command line.

giovannipizzi commented 3 years ago

Thanks for the report! It would be nice to have a way to test this - do you know if there is a way to have a Btrfs or zfs on GitHub Actions?

dev-zero commented 3 years ago

According to this GHA Community Question one should be able to create and mound a file-backed block device:

truncate -s 1g disk.img
mkfs.btrfs disk.img
losetup /dev/loop0 disk.img
mkdir disk
mount /dev/loop0 disk

... the last command depends on whether the kernel they use has btrfs-support built-in. Only tested locally...