search

abbbi / virtnbdbackup

Backup utility for Libvirt / qemu / kvm supporting incremental and differential backups + instant recovery (agentless).
http://libvirtbackup.grinser.de/
GNU General Public License v3.0
340 stars 46 forks source link

error in creating inc or diff backup #140

Closed aminhashemi1994 closed 1 year ago

aminhashemi1994 commented 1 year ago

Version used 1.9.44

Describe the bug i create a full backup with no problem. but when i does some changes in the vm and i want to create an incremental backup, it says no checkpoint found and i have to create a full backup. and when i want to create another full backup it says i have already created a full backup. i'm kind of stuck in a loop here

Expected behavior when i want to do an increamental backup, i want it to be created as expected

Hypervisor information:

Logfiles: diff backup log : [2023-10-24 09:50:08] INFO root virtnbdbackup - main [MainThread]: Backup level: [diff] [2023-10-24 09:50:08] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [7000000] [2023-10-24 09:50:08] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks. [2023-10-24 09:50:08] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1] [2023-10-24 09:50:08] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/opt/kvm_pool/HDD//checkpoints] [2023-10-24 09:50:08] INFO root checkpoint - create [MainThread]: Checkpoint handling. [2023-10-24 09:50:08] INFO root checkpoint - create [MainThread]: Diff backup: saving delta since checkpoint: []. [2023-10-24 09:50:08] ERROR root virtnbdbackup - main [MainThread]: No existing checkpoints found, execute full backup first.

full backup log :

[2023-10-24 09:51:16] INFO lib common - printVersion [MainThread]: Version: 1.9.44 Arguments: /usr/local/bin/virtnbdbackup -d 20231024-082035620420084-158_test-backup -l full -o /opt/kvm_pool/HDD/ [2023-10-24 09:51:16] INFO root virtnbdbackup - main [MainThread]: Backup level: [full] [2023-10-24 09:51:16] ERROR root virtnbdbackup - main [MainThread]: Target directory already contains full or copy backup. Workaround: Share possible workarounds, if any.

abbbi commented 1 year ago

your dif fbackup fails with:

[2023-10-24 09:50:08] ERROR root virtnbdbackup - main [MainThread]: No existing checkpoints found, execute full backup first.

your full backup failed with:

Target directory already contains full or copy

but no log was provided that suggests wether if the full backup done before was successful for not.

Something has failed before or you have an issue with your workflow. Also, the libvirt verison you are using does by default not support full/incremental backup if not enabled like described in the README, may well be that the first backup has done a fallback to copy backup because virtual machine is missing required configuration.

Please provide logfile for the FIRST backup placed in /opt/kvm_pool/HDD/

During the first full backup the backup should have created a file

/opt/kvm_pool/HDD/.cpt

in your case:

0231024-082035620420084-158_test-backup.cpt

where it stores the list of checkpoints. During the diff backup, no checkpoints are found:

No existing checkpoints found, execute full backup first.

and thats pretty strange. Please make sure you use one folder per backup set and placethe backups for different virtual machines in different folders.

virtnbdbackup -d 20231024-082035620420084-158_test-backup -l full -o /opt/kvm_pool/HDD/weekly-vm1 virtnbdbackup -d 20231024-082035620420084-158_test-backup -l inc -o /opt/kvm_pool/HDD/weekly-vm1

If you create a new full backup, rotate folders:

virtnbdbackup -d 20231024-082035620420084-158_test-backup -l full -o /opt/kvm_pool/HDD/monthly-vm1 virtnbdbackup -d 20231024-082035620420084-158_test-backup -l inc -o /opt/kvm_pool/HDD/monthly-vm1

aminhashemi1994 commented 1 year ago

i have removed the directory and created this

this is the first full successfull backup :

virtnbdbackup -d 20231024-082035620420084-158_test-backup -l full -o /opt/kvm_pool/HDD/monthly-test-backup [2023-10-25 00:51:20] INFO lib common - printVersion [MainThread]: Version: 1.9.44 Arguments: /usr/local/bin/virtnbdbackup -d 20231024-082035620420084-158_test-backup -l full -o /opt/kvm_pool/HDD/monthly-test-backup [2023-10-25 00:51:20] INFO root virtnbdbackup - main [MainThread]: Backup level: [full] [2023-10-25 00:51:20] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [7000000] [2023-10-25 00:51:20] WARNING root virtnbdbackup - setOfflineArguments [MainThread]: Domain is offline, resetting backup options. [2023-10-25 00:51:20] INFO root virtnbdbackup - setOfflineArguments [MainThread]: Backup level: [copy]. [2023-10-25 00:51:20] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks. [2023-10-25 00:51:20] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1] [2023-10-25 00:51:20] INFO root checkpoint - create [MainThread]: Checkpoint handling. [2023-10-25 00:51:20] INFO root virtnbdbackup - main [MainThread]: Local NBD Endpoint socket: [/var/tmp/virtnbdbackup.54074] [2023-10-25 00:51:20] INFO root virtnbdbackup - startOfflineNBD [vda]: Offline backup, starting local NBD server, socket: [/var/tmp/virtnbdbackup.54074.vda] [2023-10-25 00:51:20] INFO root virtnbdbackup - startOfflineNBD [vda]: Local NBD Service started, PID: [54080] [2023-10-25 00:51:20] INFO nbd client - printVersion [vda]: libnbd version: 1.6.1 [2023-10-25 00:51:20] INFO nbd client - connect [vda]: Waiting until NBD server at [nbd+unix:///vda?socket=/var/tmp/virtnbdbackup.54074.vda] is up. [2023-10-25 00:51:21] INFO nbd client - _getBlockInfo [vda]: Using Maximum Block size supported by NBD server: [33554432] [2023-10-25 00:51:21] INFO nbd client - connect [vda]: Connection to NBD backend succeeded. [2023-10-25 00:51:21] INFO root virtnbdbackup - backupDisk [vda]: Got 2805 extents to backup. [2023-10-25 00:51:21] INFO root virtnbdbackup - backupDisk [vda]: 536870912000 bytes disk size [2023-10-25 00:51:21] INFO root virtnbdbackup - backupDisk [vda]: 10272047104 bytes of data extents to backup [2023-10-25 00:51:21] INFO root virtnbdbackup - openTargetFile [vda]: Write data to target file: [/opt/kvm_pool/HDD/monthly-test-backup/vda.copy.data.partial]. [2023-10-25 00:51:21] INFO root virtnbdbackup - backupDisk [vda]: Creating thin provisioned stream backup image

[2023-10-25 00:51:47] INFO root virtnbdbackup - backupDisk [vda]: Stopping NBD Service. [2023-10-25 00:51:47] INFO root virtnbdbackup - backupDisk [vda]: Saving checksum to: [/opt/kvm_pool/HDD/monthly-test-backup/vda.copy.data.chksum] [2023-10-25 00:51:47] INFO root metadata - backupConfig [MainThread]: Saving VM config to: [/opt/kvm_pool/HDD/monthly-test-backup/vmconfig.copy.xml] [2023-10-25 00:51:47] INFO root metadata - backupBootConfig [MainThread]: Save additional boot config [loader] to: [/opt/kvm_pool/HDD/monthly-test-backup/OVMF_CODE_4M.ms.fd] [2023-10-25 00:51:47] INFO root metadata - backupBootConfig [MainThread]: Save additional boot config [nvram] to: [/opt/kvm_pool/HDD/monthly-test-backup/20231024-082035620420084-158_test-backup_VARS.fd] [2023-10-25 00:51:47] INFO root metadata - backupDiskInfo [MainThread]: Saved qcow image config to: [/opt/kvm_pool/HDD/monthly-test-backup/vda.copy.qcow.json] [2023-10-25 00:51:47] INFO root metadata - backupAutoStart [MainThread]: Autostart setting configured for virtual machine. [2023-10-25 00:51:47] INFO root virtnbdbackup - main [MainThread]: Finished successfully

and again when i want to create an incremental backup

i go to the vm and create some directories and files then i type the command sync and then i try to create the incremental backup.

as you can see in the details below : in the vm >> root@test-backup:~/test# touch asd sfg ljasd fu sj ld h sk shd f root@test-backup:~/test# ls asd f fu h ld ljasd sfg shd sj sk root@test-backup:~/test# sync

in the host : >> root@hv15:~# virtnbdbackup -d 20231024-082035620420084-158_test-backup -l inc -o /opt/kvm_pool/HDD/monthly-test-backup [2023-10-25 00:54:32] INFO lib common - printVersion [MainThread]: Version: 1.9.44 Arguments: /usr/local/bin/virtnbdbackup -d 20231024-082035620420084-158_test-backup -l inc -o /opt/kvm_pool/HDD/monthly-test-backup [2023-10-25 00:54:32] INFO root virtnbdbackup - main [MainThread]: Backup level: [inc] [2023-10-25 00:54:32] INFO root virtnbdbackup - main [MainThread]: Libvirt library version: [7000000] [2023-10-25 00:54:32] INFO root virtnbdbackup - main [MainThread]: Backup will save [1] attached disks. [2023-10-25 00:54:32] INFO root virtnbdbackup - main [MainThread]: Concurrent backup processes: [1] [2023-10-25 00:54:32] INFO root checkpoint - redefine [MainThread]: Loading checkpoint list from: [/opt/kvm_pool/HDD/monthly-test-backup/checkpoints] [2023-10-25 00:54:32] INFO root checkpoint - create [MainThread]: Checkpoint handling. [2023-10-25 00:54:32] ERROR root virtnbdbackup - main [MainThread]: No existing checkpoints found, execute full backup first.

i have added and

</qemu:capabilities> this using virt edit to the vm xml and the increamental backup capability went to last line of the xml before bedore like this :

. .

what do you suggest me to do ? do i need to upgrade my libvirt version ? i will test that too
aminhashemi1994 commented 1 year ago

also this is what i have in the fullbackup directory :

root@hv15:/opt/kvm_pool/HDD/monthly-test-backup# du -h * 528K 20231024-082035620420084-158_test-backup_VARS.fd 4.0K autostart.copy 4.0K backup.full.10252023005120.log 4.0K backup.inc.10252023005432.log 4.0K checkpoints 3.5M OVMF_CODE_4M.ms.fd 9.6G vda.copy.data 4.0K vda.copy.data.chksum 4.0K vda.copy.qcow.json 8.0K vmconfig.copy.xml

abbbi commented 1 year ago

The Virtual machine was offline during first Backup! Full Backup that is required as base for further backups Must be taken if vm is online and running, bcs only then libvirt can create Checkpoints!

setOfflineArguments [MainThread]: Domain is offline, resetting backup options. [2023-10-25 00:51:20] INFO root virtnbdbackup - setOfflineArguments [MainThread]: Backup level: [copy].

Please ensure vm is running while taking Full backup