search

our-source / restic

Restic inside docker with CRON for automated back-ups
MIT License
9 stars 12 forks source link

restic reporting 128 TiB when backing up /var/lib/docker/aufs!!!!! #3

Open rgusick opened 6 years ago

rgusick commented 6 years ago

Everything has gone smoothly setting up this container to backup my Ubuntu system. I've created an exclude file to avoid wasting time with files and directories that I don't need backed up. The problem appeared when processing /var (specifically /var/lib/docker/aufs). restic is reporting that it needs to analyze up 128 TiB of data!!!!!!! 128 TiB

The actual data:

> du -hs  /var/lib/docker/aufs  
19G     /var/lib/docker/aufs
> find  /var/lib/docker/aufs | wc -l 
696063
> du -hs  /volume1/backups/restic-repo
37G     /volume1/backups/restic-repo

Needless to say my backup for this one subdirectory (which is already in my EXCLUDE list) has effectively made my backup almost 9000x slower (19 GB vs 128 TiB) than it should be!!!!

Help!

johansmitsnl commented 6 years ago

Can you show me your exclude file or command line?

rgusick commented 6 years ago

cat ~/docker/restic/excludes.txt /data/bin/*

/data/boot/*

/data/dev/*

/data/etc/*

/data/home/*

/data/initrd.img/*

/data/initrd.img.old/*

/data/lib/ /data/lib32/ /data/lib64/ /data/lost+found/ /data/media/ /data/mnt/ /data/nxfilter/*

/data/opt/*

/data/proc/*

/data/root/*

/data/run/ /data/sbin/

/data/sources/*

/data/srv/ /data/sys/ /data/tftpboot/ /data/tmp/

/data/usr/*

/data/var/*

/data/vmlinuz/*

/data/vmlinuz.old/*

/data/volume1/ # # # /data/root/downloads/ /data/var/lib/ # # .git aufs cache ~ .cpan ## build_dir bin sbin lib log tmp .log .log.

On Tue, Oct 23, 2018 at 2:58 PM Johan Smits notifications@github.com wrote:

Can you show me your exclude file or command line?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/johansmitsnl/docker-restic/issues/3#issuecomment-432376166, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHIfgwZ-5FU1JZ3_2cTdzvBQIZsNMUxks5un2a3gaJpZM4X2Cvu .

johansmitsnl commented 6 years ago

is the exclude file available in the docker container? Can you post you compose file or command line?

rgusick commented 6 years ago

restic-jsmitsnl: restart: always image: jsmitsnl/docker-restic:latest hostname: YYYYYYYYY domainname: XXXXXXX.net container_name: restic-jsmitsnl volumes:

On Wed, Oct 24, 2018 at 1:35 AM Johan Smits notifications@github.com wrote:

is the exclude file available in the docker container? Can you post you compose file or command line?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/johansmitsnl/docker-restic/issues/3#issuecomment-432518963, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHIfnlxU2EzHU-G046ym1AET_Phk-ZSks5un_wvgaJpZM4X2Cvu .

johansmitsnl commented 6 years ago

Looking at the documentation https://restic.readthedocs.io/en/latest/040_backup.html#including-and-excluding-files should you exclude the directory like this: /data/var/lib/** with to asterisks to match all sub directories to?

In my case I don't backup / and only a the data directory. The host contains only docker and I backup only the docker mounted dirs + configs. In your case looks like ${USERDIR}/docker. And I don't exclude anything.