sourcefrog / conserve

🌲 Robust file backup tool in Rust
Other
260 stars 21 forks source link

Implementing a read-only projection of the conserve archive for Windows #240

Open WolverinDEV opened 9 months ago

WolverinDEV commented 9 months ago

Hey,
concluding my latest comment on the discussions https://github.com/sourcefrog/conserve/discussions/237 I open this PR.

Motivation

Right now it is not possible to view files which have been backupped with conserve without restoring a partial or complete versions of that particular backup. Fully restoring a specific version is mandatory in case of a complete data loss (e.g. corrupted hard drive) but inconvenient when only a few files need to be restored especially if these files are contained in different subtrees. Therefore having a feature which allows the user to easily access, view and copy only specific folders or files for specific versions is required.

Implementation

As discussed in the discussion linked above, Windows does ship an optional feature called Windows Projected File System (ProjFS).

The Windows Projected File System (ProjFS) allows a user-mode application called a "provider" to project hierarchical data from a backing data store into the file system, making it appear as files and directories in the file system.

Leveraging the ProjFS we're able to give the user a visual (and explorable) representation of all folder and files contained within the conserve backup. Leveraging the possibility of a virtual file system, the user is also able to easily view different versions of the backupped file system.

In the past few days I've implemented a rust wrapper around the Windows ProjFS API.
More details can be found here: https://github.com/WolverinDEV/windows-projfs

Pending Tasks

This PR is still in draft as the following tasks need to be completed before this feature can be shipped:

sourcefrog commented 9 months ago

I meant, a lot of dependencies, or anything that might be particularly likely to cause build breakage. For example, libssh depends on a native library and C compiler, from memory.

WolverinDEV commented 9 months ago

I get what you mean.
windows-projfs does only depend on 5 dependencies (log, parking_lot, thiserror, windows, and libloading).
As far as I know are none of them known for causing build breakage.
The windows crate is statically generated before being deployed to crate.io therefore no hidden build requirements.

The only change is I use functionality of the latest Rust stable release, hence the CI pipeline edit to 1.75.

sourcefrog commented 8 months ago

I'll come back and go line by line, but just as a general comment I would be delighted to merge this, but I would want to have some more tests, including hopefully an integration test that mounts an archive on Windows and checks that you can read from it.

WolverinDEV commented 8 months ago

I'll come back and go line by line, but just as a general comment I would be delighted to merge this, but I would want to have some more tests, including hopefully an integration test that mounts an archive on Windows and checks that you can read from it.

Hey,
yes including tests is one of the bullet points which is missing before draft completion.
Personally I've already added tests to windows-projfs but as the exam phase approaches, I got less and less time...

WolverinDEV commented 2 months ago

Hey ho,
I've successfully implemented tests :)
You're free to review now and may merge.

sourcefrog commented 1 month ago

Ack, I'll look.

sourcefrog commented 1 month ago

Thanks for adding tests! It does look like they are currently failing in CI: https://github.com/sourcefrog/conserve/actions/runs/10197526482/job/28210539695?pr=240

WolverinDEV commented 1 month ago

Thanks for adding tests! It does look like they are currently failing in CI: https://github.com/sourcefrog/conserve/actions/runs/10197526482/job/28210539695?pr=240

Yes, I had the same issue with the projfs tests. I narrowed the issue down to a resource limitation. The tests work flawlessly when reducing the number of Rust test threads (e.g., https://github.com/WolverinDEV/windows-projfs/blob/master/.github/workflows/rust.yml#L54).

I'm not quite sure what implications this might have for all the other conserve tests. Maybe execute them separately?