search

NAUbackup / VmBackup

XenServer simple backup script
228 stars 62 forks source link

:max_backups override #88

Closed VARGA-Peter closed 4 years ago

VARGA-Peter commented 5 years ago

Dear Tobias,

sorry, if I am asking an obvious question, but it isn't that clear to me. Suppose I have these VMs:

VM-1
VM-2
VM-3
VM-4
VM-5

and VmBackup.cfg has max_backups=3.

According to the comment in VmBackup.cfg the setting for max_backups can be overridden. So, when I have now these lines:

vm-export=VM-1:2
vm-export=VM-2:1
vm-export=VM-*

Do I understand it correctly that in this constellation [and order] VM-1 and VM-2 will have personalized max_backups setting as defined after the : in the corresponding line and to all VMs which match VM-* [VM-3, VM-4 and VM-5 - which have NOT been defined before explicitly] the global setting of max_backups applies?

VARGA-Peter commented 5 years ago

OK, I tested it now on a host. Please no offense, but I consider it as a bug as it is now implemented. Have a look:

Settings in VmBackup.cfg:

max_backups=2
vm-export=APA-BKP102WX:1
vm-export=APA-*

And now look at the summary:

2018/10/28 01:01:02,vmbackup.py,APA92,begin
2018/10/28 01:01:02,vm-export,APA92,begin,APA-BKP102WX
2018/10/28 01:09:19,vm-export,APA92,end,SUCCESS APA-BKP102WX,elapse:8 size:27G
...
2018/10/28 02:44:03,vm-export,APA92,begin,APA-BKP102WX
2018/10/28 02:54:16,vm-export,APA92,end,SUCCESS APA-BKP102WX,elapse:10 size:27G

So, as you can see the VM APA-BKP102WX has been processed twice. This is definitely not correct.

Could you reconsider the internal flow? From my point of view I think it can be fixed easy: Having an internal array - valid only for the current backup session - with already processed VMs so a VM is NOT processed again even when it matches any vm_export= setting.

For the moment I extended the storage space so for me the current implementation works because I commented the lines with the additional vm-export= settings.

NAUbackup commented 5 years ago

Hi, Peter: Do you get the same backup count if you run with the "preview" option? Did you try putting the order as:

max_backups=2 vm-export=APA-* vm-export=APA-BKP102WX:1

instead to see if the behavior was the same? If the same error results, I'll have to do some more debugging.

Thanks for the feedback, --Tobias

VARGA-Peter commented 5 years ago

Unfortunately, changing the order doesn't fix it: The one VM is always listed twice:

vm-export (cnt)= 5 vm-export: APA-APP231LY, APA-BKP102WX, APA-BKP102WX:1, APA-SBS201WX, APA-TS230W

NAUbackup commented 5 years ago

OK, thanks, Peter, I will then need to investigate. Thanks for the test results.

--Tobias

NAUbackup commented 5 years ago

Peter, You need to insert a period in front of your wildcard construction:

vm-export=APA-.*

I need to update the example to better point this out. This is because of the way python deals differently with wildcards when used as part of a REGEXP.

Please try this again and let me know if that then works also for you. The order of the exception does not then matter within your CFG file (vm-export=APA-BKP102WX:1 can come before or after vm-export=APA-.*).

--Tobias

NAUbackup commented 5 years ago

Here are the source code differences. capture

VARGA-Peter commented 5 years ago

Tobias, the preview result doesn't change:

vm-export: APA-APP231LY, APA-BKP102WX, APA-BKP102WX:1, APA-SBS201WX, APA-TS230W

I ran again another backup job and unfortunately nothing changed - the VM has been processed twice.

vm-export=APA-BKP102WX:1
vm-export=APA-.*
2018/10/29 22:13:24,vmbackup.py,APA92,begin
2018/10/29 22:13:24,vm-export,APA92,begin,APA-BKP102WX
2018/10/29 22:22:15,vm-export,APA92,end,SUCCESS APA-BKP102WX,elapse:8 size:27G
...
2018/10/29 23:57:01,vm-export,APA92,begin,APA-BKP102WX
2018/10/30 00:06:37,vm-export,APA92,end,SUCCESS APA-BKP102WX,elapse:9 size:27G
2018/10/30 00:06:37,vmbackup.py,APA92,end,SUCCESS,S:5 W:0 E:0
NAUbackup commented 5 years ago

Peter, Oh, the VM is processed twice -- I see. It should list only once in the preview, but even if it doesn't it sounds indeed like a bug. I will investigate further. Thank you for the feedback.

--Tobias

NAUbackup commented 5 years ago

I can reproduce this even with the "preview" option. Will have to review the logic and deal with what end up being duplicates.

NAUbackup commented 5 years ago

Still working on this. Sorry for the long delay, but lots of other things have limited the time I can spend on this.

VARGA-Peter commented 5 years ago

I know Tobias, take your time. I extended just yesterday the storage so all backups fit. I don't have any pressure.

lancefogle commented 5 years ago

This is now fixed in OnyxBackupVM

@NAUbackup if you look at the service.py code and follow from _validate_vm_list function you can see how I resolved this issue. In short, I created a list of already matched vms (vm_matched) within each list iteration that is added to when a match is found for a particular VM within the current list so future entries for that given list (i.e. vm-export, vdi-export) won't match that same VM again.

Essentially, as long as more specific matches are included in config first with this method, you will get the desired result.

Let me know if I can help at all.

NAUbackup commented 5 years ago

Thanks, Lance, that's great news. I will take a look and thanks for your efforts on this! I've been swamped with many other things meanwhile and have not had a chance to investigate, so this will be very useful.

NAUbackup commented 5 years ago

I just published release V3.24 which should address all of the issues mentioned above.