sometimes slow backup of unchanged files on NFS

Note: debugging this on a specific system right now, keeping some notes here.

Is this a BUG / ISSUE report or a QUESTION?

Unclear yet.

1.1.17

Redhat or rh-derivate.

6(?) cores, 32GB RAM

TBD

Roughly this:

borg create --compression lz4 --list --files-cache=ctime,size ... /mnt/localhdd::repo /mnt/remotenfs/

Sometimes backup slows down from the usual 100..1000 unchanged files per second (status U) to only a few per second.

Usually when run as a nightly cronjob. When trying to reproduce interactively, it is hard to reproduce.

When processing lots of unchanged files, borg is doing:

a stat call on the src file to determine ctime,size for change detection
if unchanged (U status), it will not need to read the file contents as they did not change and are already in the repo. thus there is not a lot of data volume read from the source neither there is a lot of data written to the repo.
but it will still need to read metadata like bsdflags, xattrs, acls (because we do not have them in files cache, but need them for the current backup) - this is the default behaviour.
if one is not interested in archiving these, one can omit them using options --nobsdflags --noacls --noxattrs.
id-to-name lookups to map uid / gid to username / groupname (usually fast when local, but can be slow if delegated remotely).
if one is not interested in archiving username / groupname (but only numeric uid and gid), one can omit them using --numeric-owner (borg 1.1) or --numeric-ids (borg > 1.1.x).

Other stuff coming to mind as difference between cron use and interactive use:

environment variables?
sh vs. bash? any quoting / expansion differences? do the borg excludes work correctly?

Added now --numeric-owner --noatime --noacls --noxattrs --nobsdflags to try if this resolves the issue.

Also considering monitoring with atop and doing a timestamped strace when it's running slowly.