It's normal that gzip raw targets takes a long time ? #97

Open Zardoz89 opened 8 years ago

I'm just trying btrbk with a server where we have some developer tools (Nexus, Jenkins, GitLab, FishEye), where we need a better robust backup system. Sadly, our backup "server" are shared folders of a Windows server via CIFS with DFS, that we have mount with mount.cifs , so we need to use the target raw option.

I try first with a small subvolume to see if this all works. No problem. However when I try it with the big volume (200GiB), btrbk looks like become freeze executing gzip compression over the output file. I wait like a bit more that an hour. This is normal that takes too long time ? I ended aborting it.

Config File : btrk.conf.txt

DryRun output :

sudo btrbk -c /etc/btrbk/btrbk.conf -v dryrun
btrbk command line client, version 0.23.2  (Wed Aug  3 14:59:06 2016)
Using configuration: /etc/btrbk/btrbk.conf
Creating subvolume snapshot for: /mnt/btrfs_pool/@data
[snapshot] source: /mnt/btrfs_pool/@data
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@data.20160803_1
Creating subvolume snapshot for: /mnt/btrfs_pool/@fisheye_data
[snapshot] source: /mnt/btrfs_pool/@fisheye_data
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803_1
Creating subvolume snapshot for: /mnt/btrfs_pool/@opt
[snapshot] source: /mnt/btrfs_pool/@opt
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@opt.20160803_1
Creating subvolume snapshot for: /mnt/btrfs_pool/@var
[snapshot] source: /mnt/btrfs_pool/@var
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@var.20160803_1
Creating subvolume snapshot for: /mnt/btrfs_pool/@log
[snapshot] source: /mnt/btrfs_pool/@log
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@log.20160803_1
Creating subvolume snapshot for: /mnt/btrfs_pool/@backup_etc
[snapshot] source: /mnt/btrfs_pool/@backup_etc
[snapshot] target: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803_1
Checking for missing backups of subvolume "/mnt/btrfs_pool/@data" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@data.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@data.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@data.20160803--5b2f8cf1-cf46-8d45-a554-adad5dadf108.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@data.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@data.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@data.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000007.btrfs.gz
Created 2/2 missing backups
Checking for missing backups of subvolume "/mnt/btrfs_pool/@fisheye_data" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.20160803--50ee564c-74b4-684d-9059-33fa0b466085.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000008.btrfs.gz
Created 2/2 missing backups
Checking for missing backups of subvolume "/mnt/btrfs_pool/@opt" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@opt.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@opt.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@opt.20160803--bd48bf23-d91e-b843-966c-0fae4f8c749c.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@opt.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@opt.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@opt.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000009.btrfs.gz
Created 2/2 missing backups
Checking for missing backups of subvolume "/mnt/btrfs_pool/@var" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@var.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@var.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@var.20160803--43cb72c9-5204-574e-b9e6-d82b7f69ec38.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@var.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@var.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@var.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000010.btrfs.gz
Created 2/2 missing backups
Checking for missing backups of subvolume "/mnt/btrfs_pool/@log" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@log.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@log.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@log.20160803--d1a48069-4a65-4f4f-b104-5ce5dec23ed0.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@log.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@log.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@log.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000011.btrfs.gz
Created 2/2 missing backups
Checking for missing backups of subvolume "/mnt/btrfs_pool/@backup_etc" in "/var/imagenes_dgb/infor/desarrollo03/backups/"
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.20160803--4ef02cfe-44fd-4142-8323-c526c6c26dea.btrfs.gz
Creating subvolume backup (send-receive) for: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803_1
Creating full backup...
[send-to-raw] source: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803_1
[send-to-raw] target: /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000012.btrfs.gz
Created 2/2 missing backups
Cleaning backups of subvolume "/mnt/btrfs_pool/@data": /var/imagenes_dgb/infor/desarrollo03/backups/@data.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@data.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@data.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@data.*
Cleaning backups of subvolume "/mnt/btrfs_pool/@fisheye_data": /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.*
Cleaning backups of subvolume "/mnt/btrfs_pool/@opt": /var/imagenes_dgb/infor/desarrollo03/backups/@opt.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@opt.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@opt.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@opt.*
Cleaning backups of subvolume "/mnt/btrfs_pool/@var": /var/imagenes_dgb/infor/desarrollo03/backups/@var.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@var.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@var.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@var.*
Cleaning backups of subvolume "/mnt/btrfs_pool/@log": /var/imagenes_dgb/infor/desarrollo03/backups/@log.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@log.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@log.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@log.*
Cleaning backups of subvolume "/mnt/btrfs_pool/@backup_etc": /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.*
Deleted 0 subvolumes in: /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.*
Cleaning snapshots: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.*
Deleted 0 subvolumes in: /mnt/btrfs_pool/_btrbk_snap/@backup_etc.*
Completed within: 0s  (Wed Aug  3 14:59:06 2016)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.23.2)

    Date:   Wed Aug  3 14:59:06 2016
    Config: /etc/btrbk/btrbk.conf
    Dryrun: YES

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/btrfs_pool/@data
+++ /mnt/btrfs_pool/_btrbk_snap/@data.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@data.20160803--5b2f8cf1-cf46-8d45-a554-adad5dadf108.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@data.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000007.btrfs.gz

/mnt/btrfs_pool/@fisheye_data
+++ /mnt/btrfs_pool/_btrbk_snap/@fisheye_data.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.20160803--50ee564c-74b4-684d-9059-33fa0b466085.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@fisheye_data.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000008.btrfs.gz

/mnt/btrfs_pool/@opt
+++ /mnt/btrfs_pool/_btrbk_snap/@opt.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@opt.20160803--bd48bf23-d91e-b843-966c-0fae4f8c749c.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@opt.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000009.btrfs.gz

/mnt/btrfs_pool/@var
+++ /mnt/btrfs_pool/_btrbk_snap/@var.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@var.20160803--43cb72c9-5204-574e-b9e6-d82b7f69ec38.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@var.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000010.btrfs.gz

/mnt/btrfs_pool/@log
+++ /mnt/btrfs_pool/_btrbk_snap/@log.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@log.20160803--d1a48069-4a65-4f4f-b104-5ce5dec23ed0.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@log.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000011.btrfs.gz

/mnt/btrfs_pool/@backup_etc
+++ /mnt/btrfs_pool/_btrbk_snap/@backup_etc.20160803_1
*** /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.20160803--4ef02cfe-44fd-4142-8323-c526c6c26dea.btrfs.gz
*** /var/imagenes_dgb/infor/desarrollo03/backups/@backup_etc.20160803_1--XXXXXXXX-XXXX-XXXX-XXXX-000000000012.btrfs.gz

NOTE: Dryrun was active, none of the operations above were actually executed!

Compression of 200GB (plus metadata) can take a long time, I guess this is normal. Your config looks ok too. In order to speed up things, you might give a try with the stream_compression branch, which enables more compression algorithms (pigz, pbzip2 and lzo), e.g.:

raw_target_compress  pigz
raw_target_compress_threads  8

You might also want to run btrbk --progress to get an idea on how long it will take to complete (unfortunately there is no way to see the size of the complete send-stream, which can get considerably bigger than 200GB depending on your data structure, e.g. if you have many small files).

If you run btrbk -l debug, I'll be able to help if you run into trouble.

(I'm in vacation, be prepared for delayed answers...)

Finally, I used gzip and I run it each Sunday (so, I don't need to worry much about the time that takes it). I did a test of importing a gzip image of a smaller volume on a virtual machine. 100% success :+1:

Only I need to setup all so I keep only the last backup file. We keep do a dump to backup tape of these directory (on incremental basis), so I don't need to keep older backup files beyond a week.

Now, If would be a way of getting to a old snapshot without the need of doing umount/mount cycle by hand and nearly stopping the machine, would be great.

Now, If would be a way of getting to a old snapshot without the need of doing umount/mount cycle by hand and nearly stopping the machine, would be great.

I don't understand this question. Why do you need to umount/mount to access snapshots in the first place?

I don't understand this question. Why do you need to umount/mount to access snapshots in the first place?

If I did my research correctly, to do a rollback to a previous snapshot, I need to :

rename the subvolume
rename the snapshot to the old name of the subvolumen
umount the target path and mount it again. "-o remount" don't worked on my little test.

This 3 steps, could be quickly executed by a script, but If I'm restoring, for example, the whole /var directory, I need to stop nearly all the services when I'm doing this.

If you're replacing /var, it's obvious that you have to restart all the services which access files there.

Instead of mounting a subvolume to /var, you could also have /var be a (nested) subvolume, then you could simply move it away and replace it by a backup without remounting. But the problem remains, in order to make sure that your running services close their already opened files and use the new folder (=subvolume in this case), you'll have to restart them.

digint / btrbk

It's normal that gzip raw targets takes a long time ? #97