kakra
commented
7 years ago Some more ideas:
XFS has an API to swap the extent map of one file with another file: It effectively swaps the contents while keeping hardlinks and attributes intact. This is done atomically. The idea is to create, open and unlink a tmp file (that way it's hidden from user space), then rewrite the extents into that file. A file lease can be used to detect file changes during the process and then just skip over the file. At the end, the contents are swapped and the tmp file is closed (and thus discarded from storage space). If the process is interrupted, nothing bad happens.
Btrfs even has a native API which can do the rewrite process itself, it is transparent to user space: No tmp files needed, safe to use. An interrupted process probably only leaves the file partly fragmented but still unaltered from user space pov. Btrfs supports hinting target extent size and compression. Some heuristics and/or options should be added to shake to make use of this. E.g., we could gzip files that are rarely accessed and old, and lzo files that are rarely modified but frequently accessed, and uncompress files unconditionally if a special option is supplied to cmdline.
unbrice
I understand the current approach of rewriting the file twice to keep the inode and all its attached attributes intact (xattr, stat, perms, acls). This is especially even more important for hardlinked files (those with nlink > 1) to not break the hardlinks as there seems to be no other way of keeping the hardlink (tho, I found that taking the lease on hardlinks doesn't work here).
But for non-hardlink cases we could try to redo the file from scratch, replicating all its attributes, and then rename it in place of the other to save one rewrite phase.
Additionally, this would get rid of the opportunity for shake to exit uncleanly and leave a damaged file behind (e.g. when killed by lease breaking). What do you think?