windows VSS and verify issue

[gitboogey]

Hi,

I am using zpaqfranz (v59.3) on windows, with the -vss and -verify flags (in an elvated powershell window). When using it, the shadow copy is created, then the achiving is done, and the the shadow copy is deleted. Its all fine.

My problem is, that the verify step is done by comparing with the live filesystem, which can be changed during the archiving time, like in the user's profile directory (which is one of my use cases).
So the tool validly reports that some files were chaned ("STORED HASH EA19780EF5C43DF9 VS 635656EC9CC02201 IN FILE ...").

I think that there is a logical error, becase the verifying should be done against the state in the shadow copy, and not against a filesystem, where we know, that there will be changes, and becuse of this errors.

I suggest to change in this case the working to the following sequence: create shadow copy, do the archiving, verify against the shadow copy, delete the shadow copy.

What do you think about it?

Best regards.

[fcorbelli]

Thank you for the report, but I'm afraid you are using the wrong command for what you want to do

The verify command works to remotely check (or after an extraction) the hashes of files, OR that nothing is changed during the backup (but with VSS this is normal, then -verify doest not have sense)

If you want to do a "double check" the command to use (and the corresponding switch) is test

But "-test" and "-vss" are incompatible You cannot test a -vss by reading again from filesystem, because something can (should/must) be different

I see your point, I'll think about a two-stage run First the add, then the test, then the release of the VSS

[fcorbelli]

Please try (pre-release 59.7c)

zpaqfranz update -force

then

zpaqfranz a z:\test.zpaq c:\users -vss -test

[fcorbelli]

OK, 59.7f pre-release is uploaded (zpaqfranz update -force)

Should handle -test and -verify and even -test AND -verify AND even -ssd

[gitboogey]

Hi,

thank you for the new version, which I tested a little bit. It seem to do a requested thing with -vss & -test/-verify. I found something disturbing: although the checks seems to find no errors, there is some difference reported:

zpaqfranz v59.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-05-23) franz:-xxhash -noconsole -noeta -test -verbose -vss 24/05/2024 22:17:39 VSS: starting 24/05/2024 22:17:41 VSS: end Integrity check type: XXHASH64+CRC-32 Creating prof_12.zpqf at offset 0 + 0 Add 2024-05-24 22:17:39 2.485 1.786.539.893 ( 1.66 GB) 4T (209 dirs): -m14 Non-latin (UTF-8) 5 MAX_FRAGMENT 520.192 (507.94 KB) 2.694 +added, 0 -removed. 0 starting size 1.786.539.893 data to be added 1.690.840.151 after deduplication 710.570.663 after compression 710.570.663 total size Total speed 20.46 MB/s IO buffer 1.048.576

-test on VSS Compare archive content of: prof_12.zpqf: 1 vers, 2.694 files, 23.253 frags, 148 blks, 710.570.663 bytes (677.65 MB)

2.694 in <<//?/GLOBALROOT/Device/HarddiskVolumeShadowCopy14/Users/xxxxxx/AppData/Roaming/Thunderbird/Profiles/>> Total files found: 2.694

00002694 = same Total different file size: 0 bytes 24/05/2024 22:19:29 VSS: starting release 24/05/2024 22:19:30 VSS: end releasing

HOUSTON expected 1.786.539.893, done 1.787.130.149, diff 590.256 This could be 'normal' for a VSS operation no file errors tracked 110.277 seconds (00:01:50) (with errors)

2,

zpaqfranz v59.7f-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-05-23) franz:-xxhash -noconsole -noeta -verbose -verify -vss 24/05/2024 22:20:56 VSS: starting 24/05/2024 22:20:58 VSS: end Integrity check type: XXHASH64+CRC-32 + CRC-32 by fragments Creating prof_22.zpqf at offset 0 + 0 Add 2024-05-24 22:20:56 2.485 1.786.539.893 ( 1.66 GB) 4T (209 dirs): -m14 Non-latin (UTF-8) 5 MAX_FRAGMENT 520.192 (507.94 KB) 2.694 +added, 0 -removed. 0 starting size 1.786.539.893 data to be added 1.690.840.151 after deduplication 710.570.663 after compression 710.570.663 total size Total speed 19.07 MB/s IO buffer 1.048.576

-verify on VSS

prof_22.zpqf: 1 vers, 2.694 files, 23.253 frags, 148 blks, 710.570.663 bytes (677.65 MB)

Verify hashes of one version vs filesystem (1 thread, -ssd for multithread) Scan done, preparing report...

OK XXHASH64 : 00002441 of 00002441 ( 1.66 GB hash check against file on disk)

Total hashed bytes 1.787.130.149 @ 72.597.398 B/s 24/05/2024 22:22:51 VSS: starting release 24/05/2024 22:22:51 VSS: end releasing

HOUSTON expected 1.786.539.893, done 1.787.130.149, diff 590.256 This could be 'normal' for a VSS operation no file errors tracked 114.801 seconds (00:01:54) (with errors)

Can you explain, where these diffs and errors come from? I found it interesting that in different runs I got the same diff numbers...

Btw can I request to implement the -noeta feature to the "t" (test) comand? In the logs (created by redirection) it is very annnoying.

Thank you in advance.

[fcorbelli]

Can you explain, where these diffs and errors come from? I found it interesting that in different runs I got the same diff numbers...

The difference is what there is between the beginning of the copy, and the end. Essentially, during the data scanning phase, not only the file names are recorded, but also the size of the files. Then the compression phase begins, which can take quite some time. During this time the files may change size or not be accessible at all (why? Who can tell? It should not happen, especially with VSS, but it does happen). So zpaqfranz starts out expecting, say, to have to process 1,000,000 bytes, however, it actually processes 1,000,035 bytes (because the sum of the files read has that size). So here, in the example, is the difference of 35 bytes. I could probably implement a more accurate report, with the individual files whose size varies. It would take up a little extra RAM

Btw can I request to implement the -noeta feature to the "t" (test) comand? In the logs (created by >redirection) it is very annnoying.

This should be already here. Can you please post a command line example, so I can fix? Thank you

[fcorbelli]

Just uploaded pre-release 59.7h

(in an elvated powershell window).

Running on non-admin shell try a runhigh() (not really smart, but should work on common scenarios)

[gitboogey]

Hi,

1, -noeta with test: works, is ok. It is only missing from the help (... h t).

2, testing -vss I have not seen any improvement, there are some time these "HOUSTON expected 1.786.712.866, done 1.788.129.077, diff 1.416.211 This could be 'normal' for a VSS operation" and some time not. I found some information that says that the shadow copy creation command finishes instantly, but it will take windows some time to create a shadow copy in the background. So to be safe before using it, it's state should be checked. See also https://learn.microsoft.com/en-us/previous-versions/windows/desktop/vsswmi/win32-shadowcopy

3, I found, that some combinations with -verbose like zpaqfranz_v59.7k.exe a prof_51.zpqf C:\Users -xxhash -test -verbose and zpaqfranz_v59.7k.exe a prof_7.zpqf C:\Users -xxhash -vss -test -verbose have in the eta output some empty lines, It seems to me, that there are somewhere unnecessary newlines printed.

4, checking the contents of the archives with the list command I found, that when using the -vss switch then the "base" folder is not added to the archive, but without it is is added. I think it should be the same. zpaqfranz_v59.7k.exe a prof_5.zpqf C:\Users -xxhash -test prof_5.zpqf contains "C:/Users/" but zpaqfranz_v59.7k.exe a prof_1.zpqf C:\Users -xxhash -vss -test prof_1.zpqf does not.

Best regards.

[fcorbelli]

(2) no improvement, but with -verbose you should see what files are different (3) this seems a bug (BTW -xxhash is useless, it is the default) (4) vss works in very strange and unknown ways, I'll check

So to be safe before using it, it's state should be checked

Handling vss is very painful: documentation almost absent, old, not working etc. No promises, I typically use zfs snapshots, not vss

Thanks for the report, really helpful

[gitboogey]

Hi,

I found another issue regarding VSS. I figured out, that for handling the shadow copies batch files (in the temp folder) are used. By examining the batch files, I have seen, that the command "wmic shadowcopy delete /nointeractive" is used before the creation and also for deleting it after the archiving. I think it is used for easy identification of the shadow copy to be used (because only one should be there at this time).

I have more problems with this approach: 1, All not related shadow copies are deleted, with the possibility of causing unintended harm, because we do not know, why they are there. (It can be an unrelated snapshot, restore point etc. ) 2, Because of this some antivirus / EDR solutions can flag his command as unwanted behavior (there are ransomwares, which use this technique), and quarantine the application, making it unusable.

I suggest to use another way to create and delete the shadow copies - using powershell scripting.

The following lines can be put into a batch file, and called as the shadow copy creation script:

@echo OFF
@powershell -command "$vscc= (Get-WmiObject -List Win32_ShadowCopy).Create("""C:\\""", """ClientAccessible"""); $vsci = Get-WmiObject Win32_ShadowCopy | Where-Object { $_.ID -eq $vscc.ShadowID }; Set-Content -Path $env:temp\zpqfvssID.txt -Value $($vsci.ID); Set-Content -Path $env:temp\zpqfvssDQ.txt -Value $($vsci.DeviceObject); Set-Content -Path $env:temp\zpqfvssVN.txt -Value $($vsci.VolumeName); $batcmd="""@echo OFF`r`n@vssadmin delete shadows /Shadow="""+$($vsci.ID)+""" /Quiet"""; Set-Content -Path $env:temp\zpqfvssDEL.cmd -Value $batcmd"

This script does not delete the existing copies, creates some temp files with some shadow copy properties for further processing, and creates even a dedicated shadow copy delete script (zpqfvssDEL.cmd). It can be generated/tweaked according zpaqfranz's needs, and I think it is a good foundation.

Please adapt and use it.

Thank you in advance.

[fcorbelli]

1, All not related shadow copies are deleted

I really do not care :) I am too lazy to make a "good" parser. During debug a lot of dangling VSS snapshots will remain

2, Because of this some antivirus / EDR solutions can flag ...

I really do not care, either :) (powershell can be stopped just like a script)

The real problem is the state The VSS remain in state=12 without becoming state=16

Why? I do not know

[fcorbelli]

PS ClientAccessibile does not seems the "good" type of VSS And managing VSS via gcc is really, really, REALLY complex, involving a mixed configuration with M$ stuff. Effort too big

[fcorbelli]

THIS is the problem

>> Get-WmiObject -Query "SELECT * FROM Win32_ShadowCopy" | Select-Object ID, DeviceObject, InstallDate, State

ID                                     DeviceObject                                    InstallDate               State
--                                     ------------                                    -----------               -----
{4B26569A-3C94-4DB3-A161-445AFDB0A719} \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4 20240531130959.884752+120    12

[fcorbelli]

Comparing zpaqfranz vss' (on users) with a very reliable backup software the differences are quite small

Z:\>zpaqfranz c snapshot zpaqfranz -xxh3 -ssd -checksum
zpaqfranz v59.7o-JIT-GUI-L,HW BLAKE3,SHA1/2,SFX64 v55.1,(2024-05-31)
franz:-xxh3 -checksum -hw -ssd
Dir compare (2 dirs to be checked)  ignoring .zfs and :$DATA
Creating 2 scan threads
31/05/2024 14:33:41 Scan dir |00000000| <<snapshot/>>
31/05/2024 14:33:41 Scan dir |00000001| <<zpaqfranz/>>

XXH3 |00|  32.59 GB       82.000 @    1.81 GB/s
XXH3 |01|  32.82 GB       82.021 @    1.82 GB/s

Parallel scan ended in 18.563000 s
=====================================================================================================
Free 0        14.870.953.984 (    13.85 GB)  <<snapshot/>>
Free 1        14.870.953.984 (    13.85 GB)  <<zpaqfranz/>>
=====================================================================================================
Dir  0        35.244.612.070 | Delta bytes| |      68.794|     18.56 <<snapshot/>>
Dir  1        35.245.125.104 |   501.01 KB| |      68.795|     16.50 <<zpaqfranz/>>
=====================================================================================================
Total  |       70.489.737.174| (65.65 GB)
Delta  |              513.034|                     1|files

Dir 0 (master) 35.244.612.070 (files 68.794) <<snapshot/>>
-----------------------------------------------------------------------------------------------------
Dir 1 (slave) IS DIFFERENT time 16.501 <<zpaqfranz/>>
size            35.245.125.104 (files 68.795)
excess  (not in 0) snapshot/AppData/Local/Temp/vsz_00017.bat
XXH3 different 2F54D66E7FBFAE2CB970521039CECFAE (0) vs B16E9E9D9417298724FFA709CA0BA8FD (1) snapshot/AppData/Local/ConnectedDevicesPlatform/L.utente/ActivitiesCache.db-shm
diff size             383.192 (0)            502.672 (1) snapshot/AppData/Local/ConnectedDevicesPlatform/L.utente/ActivitiesCache.db-wal
XXH3 different 8FBA100790353F40728F87EE388E6720 (0) vs F7107B1E37DCF65B01E041BE40C8A3A1 (1) snapshot/AppData/Local/Dropbox/logs/1/1-6847-6659aad8.tmp
diff size               2.302 (0)              2.474 (1) snapshot/AppData/Local/Dropbox/metrics/store.bin
XXH3 different 41C46EBEA99EEE214578FFDE7E8C96ED (0) vs B36EDB82724C92913C42C4537C0ADD54 (1) snapshot/AppData/LocalLow/Microsoft/CryptnetUrlCache/MetaData/57C8EDB95DF3F0AD4EE2DC2B8CFD4157
XXH3 different C7BE59178FC4A9F2EF5F728F77CA277A (0) vs F4899618791C709A01A470437053B41F (1) snapshot/AppData/LocalLow/Microsoft/CryptnetUrlCache/MetaData/7423F88C7F265F0DEFC08EA88C3BDE45_A30EA9B4E1BC5DBF09A8EF399E086D27
XXH3 different FFDE534351404C9EF1EC50F95D7FF2E0 (0) vs 1CD37D416630F7DC9FA7AAF991D33420 (1) snapshot/AppData/LocalLow/Microsoft/CryptnetUrlCache/MetaData/77EC63BDA74BD0D0E0426DC8F8008506
XXH3 different 9B0CB5FED7142ECE02329ED7A76CCF0C (0) vs D7DEB40A4190806F7834F4627D97E93C (1) snapshot/AppData/LocalLow/Microsoft/CryptnetUrlCache/MetaData/FB0D848F74F70BB2EAA93746D24D9749
XXH3 different 57C36DE1FBCEC2911B1ED98CCBDC461B (0) vs 4B0A80680FCE511071226CEABBD565A2 (1) snapshot/AppData/Roaming/Dropbox/GPUCache/data_1
diff size             393.216 (0)            786.432 (1) snapshot/ntuser.dat.LOG2
-----------------------------------------------------------------------------------------------------
DIFFERENT SLAVE DIRS!!
18.922 seconds (00:00:18) (with errors)

In the upcoming version I will not emit any error if getted size is > than expected one (the opposite can be for a media full)