Windows features / roadmap to victory

[enkore]

• [ ] Core functionality
  • [x] DACLs of files / folders (#1037)
  • [ ] Handle inherited ACLs some more-or-less good way (#1341)
  • [x] NTFS ADS (#1342)
  • [ ] --numeric-id should for SIDs vs. usernames like UID vs username on unixy
  • [ ] Check/ Add support for long paths (NT limit is ~32 kB)
• [ ] Advanced functionality for Windows
  • [ ] OIDs of files/dirs (<-- required to make links work)
  • [ ] SACLs of files / folders (privileges required?)
  • [ ] Mountable repos/archives, see the Dokany project (it also has a -naturally- limited FUSE compat layer) (#2316)
  • [ ] Some attributes, preferably those that can be translated from/to bsdflags (like platform.linux does it), e.g. compressed, but stuff like hidden, system etc. makes sense as well.
  • [ ] NTFS EA
  • Used by Windows CSC and at least one more system service ("appfixcache"). (And also Malware)
  • Also heavily used by the Windows Subsystem for Linux; these are essential to it.
  • Key/Value pairs, value max length 64K
  • APIs: NtQueryEaFile, NtSetEaFile, Flags: FILE_READ_EA, FILE_WRITE_EA, Structures: FILE_FULL_EA_INFORMATION
  • Note: NtSetEaFile overwrites all EAs. It's not a key-value API.
• [ ] World peace features
  • [ ] Be able to work with VSS (a user interface as simple as borg create repo::archive C: --use-vss would be ideal, and due to the nature/structure of VSS is also pretty much the only thing that makes sense) (#1204)
• [ ] Distribution
  • [ ] How to handle remote / SSH (i.e. detect/use external SSH distribution or bundle it)
  • [x] Single redistributable Pyinstaller binary
  • [x] Minor nitpick: zipping those 1000'ish files of the stdlib up into one ZIP would be nice, if possible. (#1145)
  • [x] A normal installer would be nice (#1110)
• [x] Development
  • [x] CI (#167 #1240 #1307)
  • [x] Which versions to target? I'd say XP is a pain, so let's do Vista/7+, since no one ever used Vista: 7 or later. This is basically in line to what Python for Windows does.
  • [x] Py >=3.5.1
• What else?
• This is a roadmap ticket (<-- maybe we also want a milestone here, I didn't want to spam the main tracker with a dozen tickets or so), so if there should be longer discussions about specific items needed separate tickets should be created.

[ThomasWaldmann]

About priorities: maybe first concentrate on basic functionality that a non-admin user needs to install borg and backup "normal" files and directories (owned by that user, not system files, not locked/open files), later move on to more special stuff.

[Anakonda]

• Doing DACLs atm. I cannot fetch SACL, permission denied even when running as admin - do we even need SACLs.
• Windows own "turn DACL to string" function uses only SID do we really need to use usernames.
• Python does not support XP. In theory Python does support Vista, I don't see point in supporting Vista.

[enkore]

• For fetching stuff like the SACL there is an extra security privilege, I'd have to look up which one and whether admin should have it by default; I tend to agree that the SACL is secondary and the DACL is the primary interest. I splitted the item.
  • Edit: It's SE_SECURITY_NAME, which has to be enabled on a per-thread basis. Refs
  • https://msdn.microsoft.com/en-us/library/windows/desktop/ms677926%28v=vs.85%29.aspx
  • https://msdn.microsoft.com/en-us/library/windows/desktop/ms717802(v=vs.85).aspx
  • https://msdn.microsoft.com/en-us/library/windows/desktop/aa446654%28v=vs.85%29.aspx
• I think storing both usernames and SIDs makes sense for the same reasons we store both ID and name on *nix.
• 7 or later it is.

[ThomasWaldmann]

http://alphavss.alphaleonis.com/

[Anakonda]

Interesting ...

[enkore]

COM is language independent... although you need a lot of glue code in vanilla C/C++ (~Cython). Good news, Python has comtypes, which makes COM very simple:

devmgr = comtypes.CreateObject("WIA.DeviceManager")
for di in devmgr.DeviceInfos:
   pass

... easier than d-bus ...

[enkore]

What's a bit hairy with DACLs (and SACLs) is that many of them are inherited. So if I e.g. back up something and then extract just a subtree of it then naturally the extracted subtree would only have the ACL entries set in that subtree, making the effective ACL of the extracted subtree and the original subtree different.

I have no really good ideas on how to deal with that, the Windows API has AFAIK no way of doing "dry run" calculations on ACLs, so we would have to implement that ourselves or use a library that can do ACL calculations (i.e. when extracting a subtree calculate the effective ACL of the subtree's root by looking at all parent directory ACLs and convert this effective ACL to explicit ACL entries on the subtree root).

This is of interest: https://msdn.microsoft.com/en-us/library/cc230312.aspx

[Anakonda]

Fetching unprotected ACLs should give any inherited entries too. Not sure how to turn off inheritance when restoring maybe setting them as protected.

See https://msdn.microsoft.com/en-us/library/windows/desktop/aa379573%28v=vs.85%29.aspx

[enkore]

That's probably the best way to handle it for now, although it would ofc. turn all inherited ACEs into explicit_access entries.

[anarcat]

regarding VSS support, it would be great if borg would support snapshotting filesystems before backing up files within it. it would require ordering the file set specified on the create commandline to group them by filesystem, then having some abstraction to support filesystem-specific snapshot models.

i wrote an LVM-specific one for Bup, and someone extended it to support VSS a while back. the code is here:

http://src.anarc.at/bup-cron.git/blob/HEAD:/bup_cron/__init__.py#l225

it's somewhat tightly coupled with bup-cron, but parts of it could probably be reused. unit tests will obviously be tricky there.

[anarcat]

snapshots should probably be moved to a separate feature request, if there's agreement that it's a useful feature.

[ThomasWaldmann]

I'ld rather keep snapshotting outside of borg, in a wrapper or in the (shell) script. There are a lot of different ways to snapshot.

[enkore]

The point with VSS is that it's the "one, only and true way" of snapshotting live file systems (=only NTFS) on windows. With VSS there are good technical reasons to have it within the backup application, not so much for the *nix ways of doing snapshots. I wouldn't see a "--vss" option (see start post entry re. VSS) as problematic.

At this point I feel that this discussion is a bit early, though. (IOW we should try to avoid back-and-forth discussions in this ticket)

[anarcat]

oh well, i tried. :)

[enkore]

Fetching unprotected ACLs should give any inherited entries too. Not sure how to turn off inheritance when restoring maybe setting them as protected.

See https://msdn.microsoft.com/en-us/library/windows/desktop/aa379573%28v=vs.85%29.aspx

Tried this, but getting "access denied" even in full privileged mode. I think this may not be supported for reading, only for writing (one could read the MSDN that way here).

Probably the next-easiest way(?): Collect trustees in the path from $file to $drive; build new ACL by creating explicit_access ACEs by querying the effective rights for each collected trustee, store that ACL instead.

[theChaosCoder]

I found an interesting project for Windows. It is basically FUSE for NT: https://github.com/billziss-gh/winfsp

Perhaps it is something one can use in the future...

[enkore]

Is there a comparison to dokany?

E: A bit here. Gets technical later.

[Anakonda]

When extracting archives from non-windows. How should we handle filenames that are illegal in windows EG file\name Print error and skip the file, print error and stop or try to extract with different name?

[ThomasWaldmann]

extract with valid name and print a warning.

[enkore]

Created bullet point "Advanced functionality for Windows" and moved some stuff into it (SACLs, borg mount, attributes and NTFS EA as well as NTFS object IDs).

[enkore]

NTFS EA are also used by the WSL ("Windows Subsystem for Linux") to store essential data for the Linux system (POSIX permissions and stuff). Not backing / restoring these makes the files inaccessible from Linux.

I updated the initial post accordingly.

[sinasalek]

For those of you looking for shadow copy support, Perhaps the easiest way to achieve this is using an open source project called Volume Shadow Copy Simple Client

I tried on windows 10, and it work very well. It did not need to install any dependency, it just works :)

[jdchristensen]

@sinasalek Nice. Can you share the script you use that pairs vscsc with borg?

[thinkrapido]

This thread is outdated.

The new initiative to port to windows based on the 1.2 release is at issue https://github.com/borgbackup/borg/issues/6402.

Please refer to it in the future.

We also need you to migrate the current state of the todo list to the current state.

Thank your for all of your effort.