Sync operation terminates before destination files are deleted

[MrGreg1]

Question I am using Veeam Backup and Replication software. My backup job runs a post job batch script. The script runs FastCopy command line Sync to mirror backups to another local storage location. When VBR retention runs, it deletes backups older than 2 weeks. This means that FC will delete backups from the destination that no longer exist in the source. The full backups are quite large resulting in very large file deletions.

FC and the script terminate before the all destination files are deleted. The files are eventually deleted but can take 7-10 minutes after the script has terminated. This gives a false sense of completion.

I am guessing you are using an API call to do the deletions. I suspect that your code does not wait until the file is actually deleted but just checks the request to delete was successful.

Is there a way that your code can wait for all files to be deleted before FC terminates?

[shirouzu]

FastCopy uses the DeleteFileW API for deletion. This API typically waits for the deletion to complete. If there is a file system driver that performs delayed deletion, I think there is no way for the application to wait.

[MrGreg1]

Thanks for the speedy reply. The full backups that have to be deleted during the sync total approximately 3.80 TB. There are 7 backup files. The largest one being 1.85 TB. Those are some very large files to delete.

Does the sync operation delete one file at a time making multiple calls to DeleteFileW or all files using a single call?

I do not think there is a system driver that is performing a delayed deletion.

It would appear that DeleteFileW API is not waiting for the deletion to complete.

Can you verify this?

[shirouzu]

DeleteFileW API can only delete one file at a time. https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew

File deletion in Windows occurs when all open handles are closed after setting the deletion flag. Therefore, deletion may be delayed if another process has the file open. However, there seems to be no way to detect this and determine when the last close occurred. (If there is a way to do that, please tell me in detail)

[MrGreg1]

I understand. Thanks for the explanation. I cannot imagine what process would be holding an open handle. When VBR retention runs it deletes the older backups from the repository. That is the source where FC is concerned. VBR has no knowledge of the backup copies that reside on a different drive. That is destination where FC is concerned. Is is possible that FC is holding an open handle when doing the sync comparison?

[shirouzu]

During synchronization, files are not opened when determining whether to copy or delete them.

[MrGreg1]

I understand. Either some process is holding an open handle or DeleteFileW does not wait until the file is deleted before it returns. The doc indicates that it marks the file for deletion and returns.

I do not believe there are any open handles on these backup copy files. Could it be that the large size of the files is simply taking time to delete?

[shirouzu]

I don't know, but normally it will be deleted immediately (if no other process has the file open).

[MrGreg1]

I ran a test with just one backup file. The file is 1.37 TB. I put it in the destination but not in the source. The sync deleted the file instantly. This confirms that the deletion is immediate unless there is an open handle as you stated.

Tonight is active full night for backups. I am going to observe the sync process. It will copy 7 full backups to the backup copies drive. They will sum to approximately 4 TB. It will also delete 7 full backups from the backup copies drive. They will also sum to approximately 4 TB. The result will be both drives completely in sync.

I will use Sysinternals Handle to check for open handles on the files that FC is deleting.

Will FC copy all the new backup files first, then do all the deletions?

[MrGreg1]

I observed the job early this morning. I was not able to reproduce the problem. Each folder which equates to the backups for a machine, deleted the files immediately. The batch script terminated simultaneously with the last folder processed and the backups deleted.

It must be that some process was holding an open handle when I observed the behavior we have been discussing.

Thank you for writing FastCopy! It runs circles around RoboCopy!

[shirouzu]

Thank you for your report.

Will FC copy all the new backup files first, then do all the deletions?

Repeat copying and deleting in order, folder by folder, unless "Delete before copying" is turned ON. https://fastcopy.jp/help/fastcopy_eng.htm#del_before_sync

[MrGreg1]

I like the delete before copy idea. This could help alleviate potential disk space problems in the future. I am using fcp.exe from the bath script. Is delete before copy available from the command line? If so what is the syntax?

[shirouzu]

All settings are saved in fastcopy2.ini. It is referenced by fcp.exe as well as fastcopy.exe. In other words, if you set it with FastCopy, it will also be reflected in fcp.

[MrGreg1]

Thanks for the heads up. I have added the Delete Before Copy option. I will verify this next active full night which will be next Tuesday night. I enjoyed watching your code work last night. So I will do it again. The backup copy script starts at approximately 4:30 AM and finishes approximately 6:30 AM. Makes for an early morning :)

I ran through all of the settings and doc to learn more. I am curious about Parallel Processing. I think that a separate instance(s) of FastCopy must be launched to utilize Parallel Processing. Is that correct?

My Source and Destination are local 29 TB Raid 0 Arrays.