search

borgbackup / borg

Deduplicating archiver with compression and authenticated encryption.
https://www.borgbackup.org/
Other
11.23k stars 743 forks source link

Investigate preadv2/pwritev2 flag RWF_UNCACHED for file IO. #4906

Open textshell opened 4 years ago

textshell commented 4 years ago

There are patches in flight for the linux kernel to add an additional mechanism to allow file IO without affecting file caches. The proposal uses a new flag RWF_UNCACHED for the preadv2 and pwritev2 system calls to activate this mode. Possibly this could be interesting for borg to use instead of POSIX_FADV_DONTNEED.

Points to consider: Does using this for reading make sense even if this disables read-ahead? If missing read ahead is a problem would a reader thread help?

See https://lwn.net/Articles/806980/ for details about the inflight patches.

ThomasWaldmann commented 3 years ago

I somehow can't find the minimum kernel version required for this feature.

Also, besides kernel, are there other requirements?

infectormp commented 3 years ago

from https://man7.org/linux/man-pages/man2/readv.2.html

preadv2() and pwritev2() first appeared in Linux 4.6. Library support was added in glibc 2.26.

ThomasWaldmann commented 3 years ago

@infectormp ok, that's informative about the syscalls, but the man pages do not list the minimum requirement for the RWF_UNCACHED flag.

infectormp commented 3 years ago

@ThomasWaldmann look like RWF_UNCACHED flag was introduced in kenel 5.5

ThomasWaldmann commented 3 years ago

I tried to find that in official kernel changelog, but couldn't.

infectormp commented 3 years ago

hmm... i check latest kernel source and do not find any string with RWF_UNCACHED. So look like this work unfinished yet.

infectormp commented 3 years ago

Yet another work that can help speedup walk through files https://lwn.net/Articles/843163/

ThomasWaldmann commented 3 years ago

@infectormp no sure i understand, how could borg use that?

infectormp commented 3 years ago

@ThomasWaldmann Sorry, I don't know how borg can use this. This is just for information, maybe someone can find how to use it in the future.