continuous backups

[anarcat]

it would be nice if borg would be able to run automatically in the background. and i don't mean just as a cron job, but more in the terms that liw describes here, that is that borg automatically notices file changes and backs them up transparently (ie. no need for specific calls to borg create), maybe using the fanotify/inotify.

this is also a feature that is proeminent in commercial backup software like r1soft, crashplan or Spideroak, to name a few.

this would require some thoughts given about how often we want to snapshot files, or if the concept of snapshots still makes sense in there.

[anarcat]

re. inotify vs fanotify:

• inotify: works with all sorts of events, but doesn't scale well (you basically need to setup handlers per directories)
• fanotify: scales to a whole filesystem, but doesn't handle certain events like create, move and delete

[ThomasWaldmann]

guess some experiments outside borg would be needed first to evaluate the different methods and their scalability. everything that doesn't scale has to be considered as a (unusable) toy.

[anarcat]

yeah, i wonder how the commercial solutions handle this... custom kernel module? who knows! it's proprietary! :)

[ThomasWaldmann]

a general problem with mechanisms that just continuously record change (etherpad for example) is that they create a lot of points in time, but afterwards you do not necessarily know when the "good" points in time were (when stuff was in a good, consistent, meaningful state). that's why VCS commits are much better - a human decided that now was a good point for recording the changes (and one has way less changesets compared to the "recording" approach).

[anarcat]

yeah... i don't know how exactly the commercial tools work.

on top of the "meaning" problem, you also have the problem that you potentially generate much more data... we deduplicate, but if we create snapshots everytime a logfile is appended to, for example, this could end up being wasteful even with our deduplication code...

i guess at least the "period" could be customizable...

Wikipedia has more information:

https://en.wikipedia.org/wiki/Continuous_data_protection

Notice the distinction between "near continuous" (check every N seconds, or be notified when a file is closed) vs "continuous" (every write is replicated to the backup archive).

[balek]

Box Backup has this feature. It's very useful for users' laptops and workstations, because you don't have to choose time for backup.) It would be nice to see this feature in 1.0.=)

[ThomasWaldmann]

@balek you could also do a cron job and run a backup every 3h or so.

A problem with very frequent backups is that they create a lot of archives, so you'ld have to make sure to get rid of them again rather quickly (with prune), so the total count of archives in the repo does not get too high.

Some operations take linear time [and sometimes also space] proportional to the amount of archives in the repo (e.g. check, cache sync).

So I guess, IF we ever do such a feature, some other stuff has to be improved first so this could be done in an efficient way, thus I rather don't see this for 1.0.

[ThomasWaldmann]

@anarcat the issue "way too many changes" could be solved by write-events only trigger that the respective file is included in the next backup file set (but not immediately backed up).

In specific (relatively short) intervals (e.g. every 10mins), the files in the backup file set then would be actually backed up.

Beware of race conditions.

[DrupaListo-com]

There is a sweet little tool called hcp that is free and not open source ( Freeware ), made by Idera aka r1soft and runs on linux.

http://wiki.r1soft.com/display/LTR1D/hcp https://en.wikipedia.org/wiki/R1Soft_Hot_copy

Guys that dont have btrfs and still use ext4/ext3 will find it useful cause: "Hot Copy creates point-in-time snapshots of any Linux block device"

hcp can do a read-only OR even read-write snapshots that are mounted somewhere you tell it to mount.

Each new snapshot takes several ms to perform and you could create one, do borg backup and destroy last hcp snapshot - each 5 mins... each minute / each hour...

Pros of hcp: your backups use a snapshot of your FS, Its free and works. Uses kernel module to make your ext3/4 into sth more like btrfs/zfs - in terms of snapshotting.

Cons: hcp does not do the stuff that inotify does. Also the commercial Idera Backup does sth similar - uses a kernel module to watch continuously for changes....

So the "watching for changes part" and the snapshots can be accomplished if the user has btrfs or zfs or LVM and borg uses that somehow. For normal ext4 we need to either write a new CoW kernel module and/or use existing open source efforts if any.

I'd say "borg + btrfs + very nice gui" = some kind of analog of the commercial continuous backup programs.

[DrupaListo-com]

but for ext4 users even with hcp - there is no escaping a whole FS read each new borg create - well at least "borg create" ignores files that are same size/date like rsync to skip them right ? Or each file is entirely read in full on each "borg create" ?

[jungle-boogie]

I'd personally prefer borg remain simple and bug free.

Users may like the idea of continuous snapshots, but that will mean more configurations and more scripts to write that will probably be poorly written that will cause more problems than a cron job. Additionally, how many would actually use a continuous backup option vs. time spent engineering such option?

[RonnyPfannschmidt]

instead of thinking how borg should do it, we should explicitly exclude the actual functionality, but instead consider how one could integrate borg with a different tool that generates regular fs snapshots with something like btrfs/zfs and and have borg obtain the backups from those filesystem level snapshots instead

that way borg itself would stay reasonably simple - it would just run against different paths using different paths is many orders of magnitude more simple to implement than change tracking of alive file-systems on different platforms

[DrupaListo-com]

continuous backup is 2 things:

1. snapshotting - being able to freeze the whole FS and copy it as a whole - with borg, rdiff-backup or rsync
2. watching changes in files in realtime and keeping track of deltas - so in any momment u do a snapshot you dont need to re-read the whole FS but just copy the deltas - like with btrfs send/receive

Those two are both out of scope for borg I think, too. a ) One simple way to integrate borg in a quasi-continuous backup setup is thru tools like hcp - that do the snapshotting part only. b ) Another way is to use btrfs for snapshots (p.1 above) and also btrfs send/receive (p.2 above) so you keep taking snapshots, dumping deltas from 'btrfs send' into some folder as files into subfolders and each time you do borg create using the new delta dump files' folders as borg create sources for new archives ....

Both a and b are easy to write a 10-100 line shell script for. So maybe our best path is to 1. map the possible borg integration opportunities and 2. write generic shell scripts into a external-tools-integration folder distributed with borg.... One day these scripts may become separate borg commands, so they are isolated from the rest of borg...

Or these wrapper scripts may be isolated into a separate github repo that will be called "borg btrfs wrapper" and "borg hcp wrapper" etc...

[anarcat]

well, i agree that borg should be kept simple. yet it should also accommodate different use cases. i think this use case is a reasonable user expectation, and i think it could be interesting to scale borg to that level. @ThomasWaldmann identified clear and current scalability issues that would need to be overcome before this could be implemented. those issues could also affect other use cases, such as frequent (if not continuous) backups over a long period of time, so optimising those seems like a fairly good idea anyways.

[RonnyPfannschmidt]

The massive complexity difference between doing the actual tracking and just building on top of existing snapshot tools should be keept in mind

Note that applying btrfs delta's directly is unrealistic due to the involved complexity Using btrfs send and receive as method to provide fs level entry points for Borg however is fine

However in all cases the workflow is

1. Create a location which borg can backup
2. Borg backup
3. Discard the location

Due to the involved portability complexity I strongly advise toward only dealing with 2 for the time being

[DrupaListo-com]

@anarcat have you come up with some shell script that uses inotifywait + borg create to emulate CDP - either on every file save/write/delete or every n-minutes ? Maybe we can come up with a good name for these 'borg helper scripts' or 'borg extras' and host them in some borg-extras repo ?

You know - the usual stuff goes there:

• integration with backupninja or just simple :
• pre-hooks/post-hooks realized in a shell script - for mysqdump/pgdump, checks if dest is mounted / reachable over ssh

A standartized repo for borg-related backup-ninja style scripts or simpler as in a 100 line shell wrapper scripts would be handy yes?

[RonnyPfannschmidt]

there is the brogbackup organisation, so a repo for user scripts should be doable there, please open a separate issue for that

[DrupaListo-com]

@RonnyPfannschmidt good idea. Will open.

[ghost]

https://github.com/axkibe/lsyncd

This should allow you to use Borg to execute new backups on detected changes.

[martin21]

For BTRFS there is also the possibility to use btrfs subvol find-new. Its a blazingly fast way to find all the changes between two generations of BTRFS (by generation number). Yet I think it doesn´t list any deleted. btrfs send/receive also catch deletes.

[verygreen]

Certain other filesystems provide a way to see activity on the FS. E.g. Lustre has "changelogs" that can record changes with a given granularity. I imagine implementing every such mechanism in borg itself is not a correct solution, but having a way to tell borg "here's a list of all changed files" (i.e. no need to traverse the whole namespace, just reuse the one from... last backup (I hear there's no such thing, perhaps a reference point could be supplied instead explicitly)).

This cannot be replaced with borg create ...::{now} long_list_of_changed_files because the archive needs to reference all the non-changed files too. Also it should be smart enough to understand that when a filename is given, but is missing on access - that just means the file was deleted.

Later this could be extended by the scripts to implement the "continuous backup" at a given granularity.

e.g. if I want a backup every hour (time-machine-alike), I just have this script that executes a loop with an 1 hour wait in between the iterations that does borg create with the change list (to ease up the metadata traversals and otherwise help with the system load).

[verygreen]

So I played a bit with the idea of "borg clone".

The new recreate code in master makes this real easy to implement a borg clone command that creates an exact copy, or an exact copy with some exclusions if desired. Sample of that is in https://github.com/verygreen/borg/commit/d9582b56a2b7a5eeef8493b091d9ade421bb1524 (my python skills are nonexistent, so it took me a while to wade through everything, but that should be good enough for a simple proof of concept at least).

To actually add the changed files it seems to need relatively minor changes:

1. Instead of passing new name into the recreater, pass whole opened archive (to be opened in do_clone)
2. split recreater.recreate() into two parts, so that recreate.save() could be optional.
3. add "Changed files list" into exclude list before calling recreater.recreate() - XXX depending on change-detection implementation, this might behave strangely with hardlinks I guess? How to better handle this? 3.1 we also need to make sure the changed files list only contains unique names.
4. now after recreater.recreate() we should be able to run Archiver._process on every entry from the changed list. also a big chunk of process, the one that determines which of Archive.process* methods to call based on the file type should also be part of Archive class to make it more easily callable from other places. 4.1 We actually need to modify _process to make it capable of NOT going inside of subdirectories and use this capability with our list processing.
5. Now after all of that was done - the new adjusted clone is complete and we can do recreater.write()

All in all sounds relatively simple. But does it make sense from the actual borg developers perspective?

Now to the bad news: using fanotify in a generic way does not buy us much since it does not handle creates and deletions AND it also does not handle bind mounts, so some other less generic mechanisms should probably be employed, like btrfs/zfs snapshot differences, Lustre changelogs and such (also third party tools).

And also a side note about directory sorting that I noticed while looking at all of this stuff in _process:

            if not dry_run:
                status = archive.process_dir(path, st)
            try:
                entries = os.listdir(path)
            except OSError as e:
                status = 'E'
                self.print_warning('%s: %s', path, e)
            else:
                for filename in sorted(entries):
                    entry_path = os.path.normpath(os.path.join(path, filename))
                    self._process(archive, cache, matcher, exclude_caches, exclude_if_present,
                                  keep_tag_files, skip_inodes, entry_path, restrict_dev,
                                  read_special=read_special, dry_run=dry_run)

I imagine the sort(entries) just sorts stuff alphabetically? If so, this is a pretty bad idea on a bunch of filesystems for large directories on spinning media. Affected filesystems include reiserfs, ext4 and probably many more that have static inode allocation. The problem at hand is when you sort this list in an order different than inode numbers, every lstat that you do seeks the disk to a different inode table block somewhere which is really slow. Not sorting the output is also not an option since then you get some "random" underlying hash sorting that also might not correspond to creation order. The most optimal strategy here for things with fixed inode tables is to sort by inode number (available in readdir output). For filesystems with no static inode tables (like reiserfs) I imagine that would also lead to desired effect because it still is a good proxy for create time that leads to ondisk location (earlier created files get closer to start of disk in a chunk if they were written together). More advanced COW filesystems like zfs and bttrfs - I am less sure, those might be all over the place depending on a bunch of stuff, though. I assume sort ordering would not matter there all that much.

[ThomasWaldmann]

@verygreen maybe move the inode sorting stuff you found to a separate ticket, so it can be discussed / implemented independently of this topic. It sounds good, but how much difference (if any) it makes in practice has to be seen - caching might help here, so the gain might be negligible.

[jdchristensen]

@verygreen I suspect that the gain will be negligible compared to the backup time, and not worth the complexity. As a data point, I use unison to synchronize a local file system to a remote one, and when run with a hot file system cache and no changes to propagate, it runs in 0.8s. The directory tree has 100,000 files, and that time includes scanning all of their metadata on both ends of the link and comparing against the saved state from the last time unison ran. I think borg isn't this fast, but I would rather focus on speeding this up rather than complicating the internals.

[verygreen]

@jdchristensen It really depends. It is fast on Linux with hot caches (whole tree traversal with find for ~2M files takes ~15 seconds), but not everybody is on Linux. Whole tree traversal with hot caches on my macbook (with an SSD too!) is 2m50s, that's just traversal with find, doing nothing else.

Now the Linux system in first example takes 60-70 minutes to create an incremental archive. I am not sure what contributes to it yet. There are some large log files in /var/log that I need to backup that take a long time to recompress and chunk and then discard because they are deduplicated, but I measured that separately and it takes only about 6 minutes.

Anything that shaves an hour incremental backup into a few minutes long one makes the borg more suitable as a constant backup solution for a desktop or the like that could be run pretty frequently.

So these changelogs are a good step in that direction I think. They can be started small with just a list of files to save on traversals, but if we can determine for a large file that "yes, it just grew by a few megs since the last time, so no need to rechunk/recompress preceeding 25G of data" - that would be another great one. And so on, step by step.

[verygreen]

@jdchristensen I guess I'll add that there's another usecase which is large networked filesystems where there is no ability to have local access to them, so all you've got is remote access. Traversing those is really slow, it's not possible to have "hot cache" for the entire thing, and they span petabytes to tens of petabytes nowadays and move towards exabyte territory.

See a glimpse of it in https://www.citutor.org/SITES/AContent/home/course/content.php?_cid=53 - and this is from a system that's 10 years old and that was just one of 3 partitions too in that example.

Now, of course borg is not ready for anything like that yet on many fronts, but I hope it'll get there eventually ;)

[ThomasWaldmann]

@verygreen it would be very interesting if someone could explore the current scalability limits of borg (I once did a 7.5TB test, but bigger amounts would be more interesting and also reassuring), see #216.

[enkore]

Aside from the refcounting issues the size of the hashindex itself is also limited to ~2**31-1 unique chunks. (That'd be a ~100 GB index, plus ~90 GB for the repo index and maybe 100-300 GB repo overhead). My guess is that something else blows up before that.

[verygreen]

well, the bigger the filesystem is, the more you are constrained by your single node perofrmance on multiple fronts: cpu speed (borg is single-threaded too), bus bandwidth, network bandwidth, RPC rate and so on.

The really large scale HSM systems typically allow for multiple "data movers" - i.e. nodes that are dedicated to getting data off the fs and moving it to the tape/whatever other underlying storage and back. If I have say 4PB system and I can process 2 GB/sec on my single backup node, then full initial scan takes me 2M seconds just for the data (ignoring directory scanning and such). So about 23 days for the initial scan...

Back in the real world, I have this small system I help with that I plugged into borg recently. The initial scan took:

Time (start): Fri, 2016-04-08 19:23:20
Time (end):   Sat, 2016-04-09 04:38:13
Number of files: 974458

                       Original size      Compressed size    Deduplicated size
This archive:                1.12 TB            921.23 GB             71.43 GB

This is with lz4 compression, so nothing crazy expensive and the cpu is a reasonable (but a bit dated Intel Xeon X5680 3.33GHz, 24 cores total including HT).

Subsequent archives look like this:

Time (start): Thu, 2016-04-14 00:15:04
Time (end):   Thu, 2016-04-14 01:12:39
Number of files: 940743

                       Original size      Compressed size    Deduplicated size
This archive:                1.04 TB            850.08 GB              1.33 GB

So 10 hours for 1T and more than half of that was cpu (I happened to have run it under time out of interest for the first time):

real    555m3.597s
user    321m55.629s
sys     39m19.624s

even if we only assume 350 minutes (= real cpu time for the whole thing) - that's 6 hours per 1T, so 7TB would take a bit under 2 days which is already mighty long even before we move into petabyte scale I think?

[rsyring]

@anarcat the issue "way too many changes" could be solved by write-events only trigger that the respective file is included in the next backup file set (but not immediately backed up).

In specific (relatively short) intervals (e.g. every 10mins), the files in the backup file set then would be actually backed up.

And:

https://github.com/axkibe/lsyncd

This should allow you to use Borg to execute new backups on detected changes.

FWIW, the above two ideas are close to what I would suggest. And, in the spirit of "crawl, walk, run" I'd say keep it simple at first and then potentially add complexity and cross-platform support. So, here is my idea:

1. Have a command that could be ran that would run a "file watching" service. This service would watch for file adds/changes/deletes using something like inotify. The service could potentially watch multiple paths and/or use the exclusion commands just like create.
2. This service would simply track a unique list of files that have been changed while it was running.
3. Make create aware of this service and give it an option like --watched-only that would result in create asking the watching service for the list of files that have changed. It would then iterate over this list instead of scanning the file system.
4. create then makes the watching service aware of a completed backup and asks it to "cleanup"

There are certainly issues with this approach. In particular, mapping absolute (watch) to possibly relative (create) paths. Timing between create and watch and when a file was backed up. I'm sure there are others.

However, this seems to me to be good initial approach that would add significant value by greatly reducing the diskio required for frequent backups. It could be marked "experimental" initially while corner cases are fleshed out.

[henfri]

Hello,

this is an interesting idea/discussion. In Windows, the Archive bit could be used: https://en.wikipedia.org/wiki/Archive_bit

Greetings, Hendrik

[aiso-net]

I would like to see it work with cloudlinuxs‘ version of inotify:

https://docs.cloudlinux.com/index.html?usage_and_integration.html

[ThomasWaldmann]

@aiso-net borg always does full backups, so getting infos like from inotify would not help as borg has to get stats/acls/xattrs/bsdflags from all files/dirs.

[cmiles74]

Has any work been done on @rsyring's suggestion? I use Borg daily but can't use hourly backups because of the CPU usage at scan time. Compiling a list of changes to make the next Borg backup faster sounds like it could be a solution to making frequent backups cheaper and encourage people to use them.

[rsyring]

@cmiles74 I believe @ThomasWaldmann's last comment implies that even if we had a list of the files that have changed from something like inotify, Borg always expects to do a full backup. So even if you knew ahead of time what files had changed, Borg would still have to scan all the files to complete the backup. I don't know how borg works internally, so have no idea how hard that would be to change. I'm guessing, based on Thomas' comment, it would be a significant undertaking.

[henfri]

Hello,

you can backup one file with borg. So can't you also backup a number of files (= a list of files)?

Greetings, Hendrik

[ThomasWaldmann]

it depends on what you want.

usually borg makes a full backup of whatever you specify. if you specify less, you'll get a full backup of less.

On February 7, 2018 3:51:43 PM GMT+01:00, henfri notifications@github.com wrote:

Hello,

you can backup one file with borg. So can't you also backup a number of files (= a list of files)?

Greetings, Hendrik

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/borgbackup/borg/issues/325#issuecomment-363792619

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[nealpalmer]

I'm working on a script that will do 'quick' incremental backups.

// Basically find all files (about 4x faster than borg does it), takes about 15minutes: find "backupdir" -name "" -type f -printf "%A %7s %p\n" 2> /dev/null | sort > /tmp/filelist.new.txt // find the new or modified files comm -2 -3 /tmp/filelist.new.txt /tmp/filelist.old.txt | grep -Po "backupdir/." > /tmp/filelist.diff.txt mv /tmp/filelist.new.txt /tmp/filelist.old.txt // do some mumbo-jumbo to remove any file within a CACHEDIR directory ... // Run an hourly borg backup of ONLY the new files (I'm not entirely sure of the --include-from syntax). if [ ! -s /tmp/filelist.diff.txt ] ; then borg create -include-from /tmp/filelist.diff.txt fi

This gives me a VERY small archive each hour that only contains the new files. But also to make borg archives make sense for recovering data (the hourly archives suck for this), make a full borg backup daily (it shouldn't be adding any new file data to the repo, just a new manifest). And then I set the hourly backups to be pruned to about 3 months. And keep all of the daily backups, just because I can.

I'm tempted to write an invisible app for Windows that will use the equivalent of ionotify (FindNextChangeNotification) to generate the list of new/modified files, and send them over a tcp stream to the borg client machine where I can store them until the next hourly backup (then the 'find' command goes away, and HDD access on the machine goes down to almost nothing on the hourly backups, and the backup duration is almost exactly linear with file size).

[ThomasWaldmann]

@nealpalmer be aware that this is not just about performance, but also about correctness and completeness (of changes). new items, removed items, renamed items, metadata changes (user, group, mode, xattr, acl, bsdflags changes).

[tim-seoss]

See this commentary on a session at the Linux Storage, Filesystem, and Memory-Management Summit 2018 regarding the APIs necessary for this, and also correctness (incl crash safety). If implemented, then these changes are also relevant to general incremental backups too.

[Edit - actually include the URL] https://lwn.net/Articles/755277/

[anarcat]

@tim-seoss sorry, which commentary?

[tim-seoss]

Bah, could have sworn I'd included it, sorry about that https://lwn.net/Articles/755277/

[tarjei]

@nealpalmer did you post the script somewhere?

[infectormp]

Linux Kernel 4.20 released with fanotify improvements

[infectormp]

Linux Kernel 5.1 released and now it can handle events like create, rename and delete by fanotify https://github.com/amir73il/fsnotify-utils/wiki/Super-block-root-watch

[EthraZa]

Now it looks like Borg can shine using fanotify on Kernel 5.1.

[ThomasWaldmann]

@infectormp interesting developments! but maybe not really useful for borg, see there: https://github.com/borgbackup/borg/issues/325#issuecomment-354938074

[infectormp]

@ThomasWaldmann thank you, i read your comment before :) I was looking into the fanotify code and can say fanotify catch xattr and file metadata change https://github.com/amir73il/linux/blob/7d75979ef66b0241d01fdd83ae136cfd15a3b404/include/linux/fsnotify.h#L304

[ThomasWaldmann]

@infectormp also interesting. but borg will need to have xattrs/ACLs and other metadata of ALL files, not just of the changed ones.

[infectormp]

@ThomasWaldmann fully agree with you. I currently not see how this fanotify improvements can be used inside the borg but in this topic we talk about continuous backup so i think fanotify now can be used with some external tools together with borg for backup only changed files\dirs without full scan.