Backing up a single large file requires rechunking every time

[chris8086]

The following code guarantees that the most recently modified file in any borg backup will NOT be recorded in the files cache:

https://github.com/borgbackup/borg/blob/4f82550773d85d719004472d7d142314fb023ec1/src/borg/cache.py#L606-L613

I hit the most extreme impact of this, when attempting to backup a single virtual machine disk image. Every "borg create" would read, chunk and hash this very large file, even though it had not been modified since the last run. Inspecting the repo showed that the files cache was zero bytes in size. I was hitting this bug, since entry.cmtime was not less than self._newest_cmtime, it was in fact equal to it.

I'm not familiar enough with the snapshot/granularity issues mentioned in the comment to know what the correct fix should be, but surely every file in the backup should be eligible to enter the files cache?

As workaround I have had to "touch" a dummy file and then include this in the backup, so the main large file is seen as an older file rather than the newest.

[ThomasWaldmann]

IIRC this is documented in the FAQ.

[chris8086]

Actually, it isn't. I did check the FAQ first, but it only says the information below.

It always chunks all my files, even unchanged ones!

Borg maintains a files cache where it remembers the timestamp, size and inode of files. When Borg does a new backup and starts processing a file, it first looks whether the file has changed (compared to the values stored in the files cache). If the values are the same, the file is assumed unchanged and thus its contents won’t get chunked (again).

Borg can’t keep an infinite history of files of course, thus entries in the files cache have a “maximum time to live” which is set via the environment variable BORG_FILES_CACHE_TTL (and defaults to 20). Every time you do a backup (on the same machine, using the same user), the cache entries’ ttl values of files that were not “seen” are incremented by 1 and if they reach BORG_FILES_CACHE_TTL, the entry is removed from the cache.

So, for example, if you do daily backups of 26 different data sets A, B, C, …, Z on one machine (using the default TTL), the files from A will be already forgotten when you repeat the same backups on the next day and it will be slow because it would chunk all the files each time. If you set BORG_FILES_CACHE_TTL to at least 26 (or maybe even a small multiple of that), it would be much faster.

Another possible reason is that files don’t always have the same path, for example if you mount a filesystem without stable mount points for each backup or if you are running the backup from a filesystem snapshot whose name is not stable. If the directory where you mount a filesystem is different every time, Borg assumes they are different files. This is true even if you backup these files with relative pathnames - borg uses full pathnames in files cache regardless.

[ThomasWaldmann]

https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file

[chris8086]

Thomas, you are right, my apologies. Can I suggest we add a link from "It always chunks all my files, even unchanged ones!" to this answer? As a user my main issue was the re-chunking, not the fact that one letter in the output was different.

However I don't think we have the optimal solution here. The worst filesystem timestamp granularity that I'm aware of is FAT, at two seconds. So in cases where the file being backed up was last written <=2s ago, I fully agree that the next backup should re-read the whole file.

BUT when the most recently updated file in the backup was last written four hours ago, there is no reason to exclude it from the files cache. We should have confidence that any changes to the file would be noted in the ctime.

Do you agree that it would be better to have a time threshold in this logic? I'm suggesting that if the most recent file in the backup is older than this threshold, it would be considered eligible for inclusion in the files cache.

[ThomasWaldmann]

@chris8086 can you make a docs PR for that? first one for master, after review it needs a backport to 1.1-maint.

It depends on the file size what people stumble over. If it is a small file, they won't see a perf issue, but wonder about why it is A status. If it is a big file, like in your case, you might find the perf issue and wonder why it takes so long / reads a lot of data.

The <=2s (<= timestamp granularity) is correct, but we need to consider snapshots, too.

borg does not know when or if a snapshot was made, so it can not know whether the newest visible file could get a change after the snapshot, but within granularity (result: timestamp does not change, but file is modified).

This is completely independent of when borg reads the file, you can hold that snapshot for hours and back it up afterwards and the problem does not go away by that.

We could not even safely assume that if snapshots are made, they are made right before borg begins to backup files. You could e.g. make a full snapshot and then run multiple borg create commands one after each other. So it all ends with "we do not know when a snapshot was made" and thus we can not know if we could safely add the newest file to the cache.

What we could do is add an option to ignore this special case (that's basically a "it is not borg's fault if you shoot yourself in the foot" then), but who would use this option? People still would run into this. And if you know about the issue, you can already touch a dummy file to work around it, even without such an option.

[chris8086]

I see the problem now. Is it correct to say that the crucial factor is the time between snapshots? I think the risk only occurs if the gap between snapshots is less than the filesystem timestamp granularity. Because if snapshots are further apart than that, then ctime would be updated and borg would notice.

If this is correct, do you really have users who snapshot multiple times a second and borg-create from each of them? That seems crazy to me.

I'll certainly submit the docs PR.

[ThomasWaldmann]

No, that's not correct, read the FAQ entry I linked to (again).