We're pretty good about not relying on files being kept on disk through several strategies
(btrfs cloning, hardlinking, etc.). However, when faced with unknown user code, it happens
every once in a while that such a file does get deleted (e.g. the code compiles a temporary
executable in a location that doesn't look temporary, but is also on a different fs mount than
the tracedir, so hardlinks don't work). I think for robustness, it would be useful to detect situations
when we guessed wrong and the file was deleted before recording is over. To this end, we'd
have to have some mechanism to recover the contents of any files that we considered immutable
and were subsequently deleted. My suggestion would be to keep an open fd for such files.
Then, on trace completion, we can check if the file is still linked and if not copy the data out
of the open fd. The only complication here is that some systems may have fairly low open fd
ulimits and we don't want this to interfere with regular rr operation. My suggestion would be
to just fork off a new thread, unshare(CLONE_FILES) it, raise the ulimit to maximum and then
sendmsg the fd over whenever we insert into files_assumed_immutable.
Then on trace completion, we check that those files are still there and if not make
use of the open fd to copy out the contents of the now-deleted file into the trace.
rocallahan
commented
2 years ago
We're pretty good about not relying on files being kept on disk through several strategies (btrfs cloning, hardlinking, etc.). However, when faced with unknown user code, it happens every once in a while that such a file does get deleted (e.g. the code compiles a temporary executable in a location that doesn't look temporary, but is also on a different fs mount than the tracedir, so hardlinks don't work). I think for robustness, it would be useful to detect situations when we guessed wrong and the file was deleted before recording is over. To this end, we'd have to have some mechanism to recover the contents of any files that we considered immutable and were subsequently deleted. My suggestion would be to keep an open fd for such files. Then, on trace completion, we can check if the file is still linked and if not copy the data out of the open fd. The only complication here is that some systems may have fairly low open fd ulimits and we don't want this to interfere with regular rr operation. My suggestion would be to just fork off a new thread,
unshare(CLONE_FILES)it, raise the ulimit to maximum and then sendmsg the fd over whenever we insert into files_assumed_immutable. Then on trace completion, we check that those files are still there and if not make use of the open fd to copy out the contents of the now-deleted file into the trace.