search

HCL-TECH-SOFTWARE / domino-backup

Configuration files for HCL Domino Backup integration with other backup solutions.
https://opensource.hcltechsw.com/domino-backup/
Apache License 2.0
5 stars 4 forks source link

Windows server backup trigger domino VSS writer even when domino data directory not in scope for backup #14

Closed Smarter-repo closed 1 month ago

Smarter-repo commented 10 months ago

Environment: Domino 12.0.2 FP1 running on WIndows server 2022. Veeam backup & replication as backup system

Issue: Doing a bare metal recovery backup of the Windows 2022 server using the built in backup. Pasted Graphic

Domino data directory resides on drive E: and are not part of the scope for the windows backup! notes.ini 

NotesProgram=C:\Domino
Directory=E:\Domino\Data

The windows backup also trigger a domino snapshot, and seen in dominobackup.nsf with forever pending state. Pasted Graphic 2

As no domino backup actually takes place when triggered by windows server backup the list of Backup times: for a database get incorrect. And a bigger problem is that DELTA files get created and will eventually fill up the disk.

Expected behavior would be if domino data directory is out of scope, the Domino VSS writer should not signal interest. As stated in HCL documentation step 3

Console showing the above two events

When Backup initiated normal by Veeam

[0B18:0004-377C] 2023-12-11 11:38:29   BackupVSS: Domino Agent VSS Freeze Event received
[0B18:0004-377C] 2023-12-11 11:38:29   BackupVSS: backup command line: Binary path: [C:\Domino\nbackup.EXE] Param: [-vss]
[3B04:0002-0610] 2023-12-11 11:38:30   Backup: Domino Database Backup 
[3B04:0002-0610] 2023-12-11 11:38:30   Backup: Started 
[3B04:0002-0610] 2023-12-11 11:38:30   Backup: Pruning backups 
[3B04:0002-0610] 2023-12-11 11:38:31   Backup: BackupNode: [t-dom2], BackupName: [default], Translog Mode: [ARCHIVE],  Backup Mode: [SNAP] 
[3B04:0002-0610] 2023-12-11 11:38:31   Backup: LastBackupTime: 2023-12-11 11:31:19 
[3B04:0002-0610] 2023-12-11 11:38:31   Backup: Starting backup for 655 database(s) [*SnapShotMode*] 
[3B04:0002-0610] 2023-12-11 11:38:34   Backup: Ready for VSS Snapshot 
[0B18:0004-377C] 2023-12-11 11:38:35   BackupVSS: DominoVSSWriter::OnFreeze Domino in Freeze Mode after 4 seconds!
[12B0:0002-0878] 2023-12-11 11:38:36   Index update process shutdown
[3B04:0002-0610] 2023-12-11 11:38:37   Backup: VSS Post Snapshot status reached after 2 seconds! 
[3B04:0002-0610] 2023-12-11 11:38:37   Backup: Capturing changes in 655 database(s) [*SnapShotMode*] 
[0B18:0004-377C] 2023-12-11 11:38:37   BackupVSS: Sending unfreeze to Domino Backup
[3B04:0002-0610] 2023-12-11 11:38:38   Backup: Applied 886,9 KB to Backup File [C:\backupvss_snapshot\Domino\Data\xxxxxxxx.nsf] 
[3B04:0002-0610] 2023-12-11 11:38:40   Backup: Applied 1,3 KB to Backup File [C:\backupvss_snapshot\Domino\xxxxxxxx.nsf] 
[3B04:0002-0610] 2023-12-11 11:38:42   Backup: Applied 3,1 KB to Backup File [C:\backupvss_snapshot\Domino\Data\xxxxxxxx.nsf] 
[3B04:0002-0610] 2023-12-11 11:38:51   Backup: Applied 344,0 Bytes to Backup File [C:\backupvss_snapshot\Domino\Data\xxxxxxxx.nsf] 
[3B04:0002-0610] 2023-12-11 11:39:08   Backup: Delta file apply completed. 
[3B04:0002-0610] 2023-12-11 11:39:09   Backup: VSS Snapshot Backup finalized 
[0B18:0004-377C] 2023-12-11 11:39:09   BackupVSS: DominoVSSWriter::OnPostSnapshot Post backup changes applied after 29 seconds.
[3B04:0002-0610] 2023-12-11 11:39:27   Backup:  
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: --- Backup Summary --- 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Previous Backup  : 2023-12-11 11:31:19 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Start Time       : 2023-12-11 11:38:31 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: End Time         : 2023-12-11 11:39:09 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Runtime          : 00:00:38.31 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup:  
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: All              :   655 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Processed        :   655 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Excluded         :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Pending Compact  :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Compact Retries  :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Backup Errors    :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Not Modified     :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Delta Files      :     0 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Delta applied    :     4 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup:  
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Total DB Size    :    95,8 GB 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Total DeltaSize  :     0,0 Bytes 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Data Rate        : 2 579,9 MB/sec 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: --- Backup Summary --- 
[3B04:0002-0610] 2023-12-11 11:39:27   Backup:  
[3B04:0002-0610] 2023-12-11 11:39:27   Backup: Finished

When initiated by windows server backup 

[2BC4:0002-384C] 2023-12-11 11:21:54   Backup: Ready for VSS Snapshot 
[1260:0004-3B6C] 2023-12-11 11:21:54   BackupVSS: DominoVSSWriter::OnFreeze Domino in Freeze Mode after 2 seconds!
[2BC4:0002-384C] 2023-12-11 11:21:56   Backup: VSS Post Snapshot status reached after 2 seconds! 
[2BC4:0002-384C] 2023-12-11 11:21:56   Backup: Capturing changes in 655 database(s) [*SnapShotMode*] 
[1260:0004-3B6C] 2023-12-11 11:21:56   BackupVSS: Sending unfreeze to Domino Backup
[2BC4:0002-384C] 2023-12-11 11:21:57   Backup: VSS Snapshot Backup finalized 
[1260:0004-3B6C] 2023-12-11 11:21:57   BackupVSS: DominoVSSWriter::OnPostSnapshot Post backup changes applied after 0 seconds.
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup:  
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: --- Backup Summary --- 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Previous Backup  : 2023-12-11 09:01:07 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Start Time       : 2023-12-11 11:21:52 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: End Time         : 2023-12-11 11:21:57 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Runtime          : 00:00:05.78 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup:  
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: All              :   655 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Processed        :   655 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Excluded         :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Pending Compact  :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Compact Retries  :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Backup Errors    :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Not Modified     :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Delta Files      :     2 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Delta applied    :     0 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup:  
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Total DB Size    :    95,8 GB 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Total DeltaSize  : 1 018,0 Bytes 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Data Rate        : 19 321,6 MB/sec 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: --- Backup Summary --- 
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup:  
[2BC4:0002-384C] 2023-12-11 11:22:16   Backup: Finished
Daniel-Nashed commented 10 months ago

If you include a disk into a VSS snapshot, how should the Domino Backup VSS Writer know a backup to the NSF disk is not intended?

I don't see that this is anything a VSS application could handle differently. I don't know which Windows backup you are using, but if this leverages VSS and includes the Domino data disk, Domino has to respond. I don't see any problem here on the Domino server side.

Can you please explain in detail how Domino should distinct between the snapshots and not being triggered in this case.

Thanks

Smarter-repo commented 10 months ago

Thanks for your response Daniel!

Will try to give some more info, Windows server backup is a server feature included but need to be enabled in Windows 2022 image

In our case we use this feature to do a bare metal backup of the windows machine (to be able to quickly restore the base OS in case of disaster). This do not have the domino data catalog in scope (which i located on E: volume) Windows server backup indeed use VSS, but as we do not backup volume E: and our belief was that Domino wouldn't show any interest?

Clip showing the items being backed up, bare minimum needed to boot the machine and restore the Domino data via Veeam. no E: in the list: Pasted Graphic 3

Daniel-Nashed commented 10 months ago

OK this needs more research. I don't understand why you are using a simple tool like Windows backup if you have Veeam.. But that's your choice. And I have tested with both just taking a backup of disk C:\ and Domino data was on Disk E:\ The data in backup in both cases was just the data on C:\ I did backup, but the backup was triggered on Disk E:.

Domino Backup VSS returns only what it is interested in when the OnIdentify is called. But someone VSS still calls Domino for a backup with OnPrepareBackup and the following events. this should not happen if only disk C:\ is requested for backup.

Those events should not show up after the OnIdentify phase. This needs a deeper analysis. What should happen is that VSS compares the disk and path we care about and see that it's not part of the requested data to be backed up.

> BackupVSS: DominoVSSWriter::OnIdentify called
> BackupVSS: DominoVSSWriter::OnIdentify called
> BackupVSS: DominoVSSWriter::OnPrepareBackup called
> BackupVSS: DominoVSSWriter::OnPrepareSnapshot called
> BackupVSS: DominoVSSWriter::OnFreeze called
[3600:0004-2F60] 11.12.2023 17:28:26   BackupVSS: Domino Agent VSS Freeze Event received
Smarter-repo commented 10 months ago

Should I initiate a ticket with HCL? Or is this something you will drive?

Daniel-Nashed commented 10 months ago

We will drive this from our end. We need to first double check in detail on HCL side and post a SPR once fully reproduced. This would be a core Domino VSS issue.

Daniel-Nashed commented 10 months ago

HCL now has reproduced the issue. here is the SPR number and short description:

SPR #DDCRCYGC6W - Windows server backup trigger domino VSS writer even when domino data directory not in scope for backup.

From my tests I have seen that Domino is not part of the snapshot. but it is triggered. that's what needs to get addressed. But what I don't see is that the Domino disk is in the snapshot.

If you see that in your case, please explain exactly what you are experiencing. We have tested with Veeam and also the simple Windows backup and also diskshadow to see what is causing the freeze trigger.

Smarter-repo commented 10 months ago

@Daniel-Nashed Have the same experience, Windows server backup trigger domino VSS writer but volume not included in snap when checking with vssadmin list shadows

But as domino was triggered, the backup shows up in dominobackup.nsf and make the overview of taken backups messy and restore harder as the perceived backup is presented as a selectable backup. Thanks Daniel for taking this forward 👍

Smarter-repo commented 6 months ago

Hi, I know a SPR have been created and this may be the wrong place to expect updates! But I go ahead anyway :-), do we have an expected timeframe for a fix?

Daniel-Nashed commented 6 months ago

This is the right place to ask and I was about to post as soon 14.0 FP1 ships. The SPR is fixed in 14.0.1 and is back ported to 14.0 FP1.

Here is the official schedule.

Daniel-Nashed commented 1 month ago

The issue is fixed in Domino 14.0 FP2. Please update to 14.0 FP2 IF1 which has been released yesterday which is fixing a regression in 14.0 FP2 in another area.