Restores corrupted files on Windows

[XBagon]

I tried this on Windows and creating a backup of a folder seemed to work flawlessly. When I tried restoring it gave me a lot of these errors:

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: The process cannot access the file because it is being used by another process. (os error 32)', C:\Users\XBagon\.cargo\registry\src\github.com-1ecc6299db9ec823\rustic-rs-0.5.3\src\commands\restore.rs:382:68

After the restore finished, I diffed the original folder and the restored one and it reported some differences on some video files, which were visibly corrupted.

[aawsome]

Thanks a lot for reporting!

The error seems to come from parallel access to files-to-restore. There are two possible reasons: a) You did access the file parallel to the restore process b) rustic does access the file in parallel, which seems to work on Linux. Maybe that doesn't apply to Windows and we need more special treatment here.

Can you judge if a) is possible or can you rule that out?

[ducalex]

Rust by default sets dwShareMode correctly so it shouldn't matter if rustic accesses the same file in parallel. I've come across two things that often cause this issue on Windows (locking a file momentarily right after it's been created/modified):

• Windows Defender (will do it for most files most of the time)
• Windows Explorer if it's open to the destination (thumbnail generation)

I'm aware of a few solutions:

• Keep one handle to the file around until the end (possibly with a process lock), then write_at() is free to do whatever
• Just retry .open() after a short delay (which seems like a good idea anyway?)
• Add Windows Defender exclusions for the destination and/or rustic.exe (which would also speed up all operations)

[aawsome]

@ducalex Thanks for your reply and suggestions! I wasn't aware of the Rust internals but good to know that in principal it should work.

Actually I think we should do all:

I'm aware of a few solutions:

* Keep one handle to the file around until the end (possibly with a process lock), then write_at() is free to do whatever

This is actually what I kind of plan in the long term, see also https://github.com/rustic-rs/rustic/discussions/629#discussioncomment-5796845. Keeping files open and holding file handles could also possibly increase performance and reduce/remove file fragmentation.

* Just retry .open() after a short delay (which seems like a good idea anyway?)

Good idea, I think this should be implemented independently and pretty fast.

* Add Windows Defender exclusions for the destination and/or rustic.exe (which would also speed up all operations)

Is this something rustic could do? Or can you give an short How-To (which we can add to the docu once it is ready, see #302 )? Sorry, I can't judge if this is trival - I didn't use Windows for a long time...

[ducalex]

* Add Windows Defender exclusions for the destination and/or rustic.exe (which would also speed up all operations)

Is this something rustic could do? Or can you give an short How-To (which we can add to the docu once it is ready, see #302 )? Sorry, I can't judge if this is trival - I didn't use Windows for a long time...

Exceptions can be added via powershell or by writing to the registry, both can be done by rustic I imagine but you'd need to run it as administrator. Perhaps it's best to just document how to do it manually:

Open an elevated powershell and run Add-MpPreference -ExclusionProcess "rustic.exe".

There's a mention in the restic FAQ but I don't think the exclusion is documented anywhere (perhaps in the forum?).

Edit: There's a few mentions in the forum, most threads refer to that thread: https://forum.restic.net/t/windows-defender-causes-10x-slowdown/925.

[aawsome]

If this is because a parallel running job (e.g. Windows Defender) accessing the file, note that the rustic restore is "resumable" in the sense that it detects already restored contents and only needs to restore the missing contents if you re-run rustic restore.

[XBagon]

So I tried a few things. I created a new backup, restored it again and had the same problem again with some video files.

Open an elevated powershell and run Add-MpPreference -ExclusionProcess "rustic.exe"

I followed this though before doing all that and ensured I didn't have the files open myself anywhere, and I'm pretty sure I don't, as it affected multiple different video files, I wasn't watching any video and didn't have the folder open in explorer or elsewhere, but I guess that's always hard to completly prove. So then I tried restoring again it did repair the files to my surprise? Also it only restored around 30MB to repair multiple files a lot larger than that, so seems to work pretty efficiently :). Cool program you have there 🥳.

I guess now the question for me is if I find this system reliable enough to actually use it for backups :P I'm looking for a nifty backup solution on Windows for so long, I might try it 😅. It seems to mostly affect large files, I guess I could try to work around those for the moment.