optimize to store LVM snapshot differences

[ThomasWaldmann]

Had an interesting discussion at 35c3 and the idea came up to add a fixed blocksize chunker for LVM LE size (4MiB? And configurable also to other block sizes, of course.). We have that now in borg 1.2, see #4238.

It would be possible then to ask LVM for the changed LEs between 2 snapshots and efficiently just add the changed LEs to a borg repo:

• do LVM snapshot lvm-ID1
• borg backup snapshot to archive repo::arch-ID1:lvm-device-ID1
• work work work
• do LVM snapshot lvm-ID2
• ask LVM for the snapshot difference D = {LEs ID2} - {LEs ID1}
• fetch chunk id list for arch-ID1 (could use the files cache or just fetch from the repo / archive metadata)
• update the repo, storing the changed LEs from D (nice and efficient: only has to read all changed LEs as all other are still the same as in ID1 archive).
• update the chunk id list with the changed LEs/chunks from D
• create an item with the updated chunk id list to repo::arch-ID2:lvm-device-ID2

[ThomasWaldmann]

Related:

https://github.com/QubesOS/qubes-issues/issues/858, especially https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-262256101

https://github.com/tasket/sparsebak

update: https://github.com/tasket/wyng-backup

[srd424]

Have you ever come across the dm-era device mapper target? Looks relevant: https://web.archive.org/web/20160811124312/http://blog.rackcorp.com/2016/03/

[ThomasWaldmann]

Interesting. But (OMG), arrays of bits (ints) as XML...

[srd424]

I don't know if the output from era_invalidate is any saner....

On Wed, 7 Aug 2019 at 15:41, TW notifications@github.com wrote:

Interesting. But (OMG), arrays of bits (ints) as XML...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/borgbackup/borg/issues/4363?email_source=notifications&email_token=AB4TJUR3E2ZRW7XTXSZRV2LQDLNJFA5CNFSM4GXMGDKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YUFMA#issuecomment-519127728, or mute the thread https://github.com/notifications/unsubscribe-auth/AB4TJUQ2YINN5MYTTDALRKDQDLNJFANCNFSM4GXMGDKA .

[srd424]

https://github.com/mpalmer/lvmsync might be worth a look too - seems to be focused on syncing images using snapshots..