Native Windows port

[ThomasWaldmann]

If borg could run directly on a normal Windows installation (via cmd, not requiring cygwin or other major runtime support), it could be useful for windows machines that do not have cygwin installed.

Notes:

Doing a native windows port is likely quite some work over a longer time, due to all the differences between windows and UNIX-like operating systems. Whether it can be done in a pretty (regarding to source code changes) and good working (regarding to practical use) way has to be seen.

Because of that, it will have to live in a separate branch of the repository at first (which should be regularly updated from master branch, so it is kept up-to-date and without potential merge conflicts).

At some time in the future, that branch might or might not get merged into master.

---

:moneybag: there is a bounty for this

I opened a (first) bounty for this - if you would like to see this happen, feel free to support this or a future, related bounty.

The goal and scope of this bounty is:

• have a borg codebase that is able to run on native windows (without cygwin or similar runtime requirements)
• working basic borg commands: init, create, list, extract, delete
• working unit tests (failures in some of them that test stuff not in scope of the bounty are acceptable)
• initial documentation for developers and users (how to set up dev environment, how to install on a user system)
• not required: encryption, special windows file attributes support (like ACLs, xattrs), VSS support, installer

[ThomasWaldmann]

See PR #947 (has been merged into "windows" branch now).

Superceded by: #986 (has been merged into "windows" branch now).

[ThomasWaldmann]

Thanks to @Anakonda for doing the native windows port!

The basics work, but it will for sure need more testing, fixes and enhancements - please file tickets for anything that shows up (make sure you mention "windows branch" for problems only happening in that branch).

[henfri]

Hello,

I would like to try borg under Windows. I am aware of the standalone installer using cygwin. But for the project, I think testing of the native windows port would be useful, don't you think?

Is there any documentation on the native port? E.g. what python packages are needed. For another project I use cx_freeze to freeze to create an executable. That would certainly help on this respect. What do you think?

Regards, Hendrik

[henfri]

Hello,

I just saw that an executable is already created here: https://github.com/borgbackup/borg/blob/windows/deployment/windows/buildwin32.py

That's good. Before deploying borg as native executable, an installer would be even better, but for testing, it is not needed I think.

If you would like people to test the native windows, I think that a compiled version of this branch would be needed. Is that available from somewhere?

Regards, Hendrik

[Anakonda]

@henfri latest version is available here https://ci.appveyor.com/project/borgbackup/borg/build/artifacts

[Neurrone]

@Anakonda is this a native windows port, or does it install sigwin?

[henfri]

Cygwin, you mean? By the looks it is the native built and does not need cygwin

Greetings, Hendrik

[Neurrone]

@henfri thanks. Will try it, curious to see how it stacks up against rclone and duplicati

[henfri]

Hello,

no, it does not run. It is missing ctypes.wintypes:

from .windows import acl_get, acl_set File "src\borg\platform\windows.pyx", line 13, in init borg.platform.windows (src/borg/platform\windows.c:8460) ImportError: No module named 'ctypes.wintypes'

Greetings, Hendrik

[Anakonda]

Here is working one: https://www.dropbox.com/s/tolumpufc179j77/borg.zip?dl=0 I will do pull request later.

[henfri]

Thanks. That runs.

But I get an error message doing

borg list "borg@openmediavault:/srv/_dev_disk_by-id_ata-ST5000DM000/borgbackup Using a pure-python msgpack! This will result in lower performance. Could not find supported ssh client. Supported clients are ssh and plink.

So, I suppose that I need "ssh" installed somewhere? But 'what' ssh and where?

Greetings, Hendrik

[Anakonda]

You are trying to backup to a server? Where did you get that address? Borg uses protocol called ssh to communicate with server so you need ssh client. Windows version support openssh (ssh) and putty (plink)

[henfri]

Hello,

yes. The adress is the one that I use for backing up my linux machine.

I will try plink. Before I ask more questions: Is there a documentation for the specifics of the windows version?

Greetings, Hendrik

[henfri]

Hello,

I tried plink now.

AssertionError: cleanup happened in Repository.del Local Exception. Traceback (most recent call last): File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 2287, in main exit_code = archiver.run(args) File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 2220, in run return args.func(args) File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\archiver.py", line 83, in wrapper repository = RemoteRepository(location, create=create, lock_wait=self.lock_wait, lock=lock, args=args) File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 184, in init version = self.call('negotiate', RPC_PROTOCOL_VERSION) File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 272, in call for resp in self.call_many(cmd, [args], **kw): File "C:\Users\Hendrik Friedel\Downloads\borg1\bin\borg\remote.py", line 324, in call_many raise Exception('FD exception occurred') Exception: FD exception occurred

Platform: Windows LaptopHendrik 10 10.0.14393 AMD64 Intel64 Family 6 Model 69 Stepping 1, GenuineIntel Borg: 1.1.dev0-201-g7b37b16d Python: CPython 3.5.3 PID: 16908 CWD: C:\Users\Hendrik Friedel\Downloads\borg1 sys.argv: ['borg', 'list', 'borg@openmediavault:/srv/_dev_disk_by-id_ata-ST5000DM000/borgbackup'] SSH_ORIGINAL_COMMAND: None

I can also try openssl. Can you point me at a download link? There seem to be several ports.

Greetings, Hendrik

[Anakonda]

There is bug in windows version, You need to authenticate via ssh key. See some guides how to set that up. As far as I know only openssh builds for windows are for cygwin and msys.

[henfri]

Hello,

I did my first backup from Windows now. Some notes: Make sure you can connect from putty to your server using public-key. I used a dedicated user ("borg") on the server. I created the key with puttygen.exe

In putty go to settings->connection->SSH->auth --> private key for authorization and select the private key that was created by puttygen. Add the public key to /home/borg/.ssh/authorized_keys

Test the connection from putty. No password should be needed. Now test the connection with plink: plink borg@server ls should give a directory listing from the server.

If this runs, you are set.

[henfri]

Hello,

would you be willing also to provide an 32bit executable?

Regards, Hendrik

[henfri]

... and with the latest Version, please as I am also suffering from this: https://github.com/borgbackup/borg/issues/3192

[henfri]

Hello,

I have used this version of Borg for a while now. I had in the past not seen errors (but I was running it in Background). But now I continiously get:

FD exception occurred

Local Exception.
Traceback (most recent call last):
  File "C:\Program Files\borg\bin\borg\archiver.py", line 95, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 311, in do_create
    create_inner(archive, cache)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 287, in create_inner
    archive.save(comment=args.comment, timestamp=args.timestamp)
  File "C:\Program Files\borg\bin\borg\archive.py", line 378, in save
    self.cache.add_chunk(self.id, Chunk(data), self.stats)
  File "C:\Program Files\borg\bin\borg\cache.py", line 386, in add_chunk
    self.repository.put(id, data, wait=False)
  File "C:\Program Files\borg\bin\borg\remote.py", line 405, in put
    return self.call('put', id_, data, wait=wait)
  File "C:\Program Files\borg\bin\borg\remote.py", line 272, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "C:\Program Files\borg\bin\borg\remote.py", line 324, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\borg\bin\borg\archiver.py", line 2287, in main
    exit_code = archiver.run(args)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 2220, in run
    return args.func(args)
  File "C:\Program Files\borg\bin\borg\archiver.py", line 95, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "C:\Program Files\borg\bin\borg\remote.py", line 209, in __exit__
    self.rollback()
  File "C:\Program Files\borg\bin\borg\remote.py", line 385, in rollback
    return self.call('rollback')
  File "C:\Program Files\borg\bin\borg\remote.py", line 272, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "C:\Program Files\borg\bin\borg\remote.py", line 324, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

How can I further debug this issue?

Greetings, Hendrik

[ThomasWaldmann]

@henfri often it is a very fundamental problem with ssh not working.

[henfri]

Hello Thomas,

wouldn't it be good to make the output a bit more clear and even isolate the problem further in the output?

What's surprising: The error happens after a long time. I am able to backup successfully if I select smaller folders.

It's a bit of a locked situation.. I cannot do anything, I fear.

Any Ideas?

[ThomasWaldmann]

Check if you ssh works if you invoke it with similar params manually.

Not sure how we can do better as the failure is in ssh.

[anarcat]

so what's the status of the Windows port? the install docs refer to the Linux subsystem or Cygwin but do not seem to refer to this .zip build... was that intentionally removed? or is that code still sitting only in a feature branch?

i may need to work on windows for a client and I'm wondering what the best way to go would be - should i try that windows branch? or cygwin?

[enkore]

• it's still sitting largely unmaintained in a feature branch (by now I think the whole idea of doing that development in a separate branch was probably misguided and doomed to fail)
• anakonda did some great work but there are pretty much exactly zero people around borg who have both the expertise and the willingness to do windows development

disclaimer: i'm probably not the best person to give answers to "what's the current status of X in borg?" any more, since I don't keep an eye on borg development any more.

[Anakonda]

@anarcat depends what you need: windows branch tends to be bit outdated, although it does have some windows specific features. see #983 Also doesn't cygwin version use cygwin paths, windows branch uses native paths.

[henfri]

Hello,

it's still sitting largely unmaintained in a feature branch (by now I think the whole idea of doing that development in a separate branch was probably misguided and doomed to fail) That is in line with my perception, but I am no expert.

Recently anaconda did merge the -then latest- master into the windows branch. I think that it would really be good to merge the branches, before they diverge again.

disclaimer: i'm probably not the best person to give answers to "what's the current status of X in borg?" any more, since I don't keep an eye on borg development any more.

I do use borg under windows. It works. There are things that could and should be improved though.

Greetings, Hendrik

[henfri]

Hello, coming back to this:

wouldn't it be good to make the output a bit more clear and even isolate the problem further in the output? What's surprising: The error happens after a long time. I am able to backup successfully if I select smaller folders. It's a bit of a locked situation.. I cannot do anything, I fear.

Check if you ssh works if you invoke it with similar params manually. Not sure how we can do better as the failure is in ssh.

But doesn't ssh report an error message, that could be printed? Currently, we just see Exception: FD exception occurred

This message does not even say that there is an error in the ssh connection.

I should clarify: Generally the connection works and I can do backups successfully. But sometimes, I get this Error. So it is hard to reproduce this (and we do not know if this error would come up, if no data is transferred.

Anyway: What are 'similar commands'? What exactly should I test?

Greetings, Hendrik

[Unkn0wn-MDCLXIV]

I'd like to create a GUI once the program is working natively on windows. I am able to compile the win32 executable with some tweaks to the current build script (_sysconfigdata_m_win32_.py does not exist), but once I am trying to run the exe on windows, it says

Failed to import the site module

ImportError: No module named 'site'

Edit: When using the precompiled binary linked here, I am not abled to create a borg backup. It fails after a few seconds with the output

File "F:\Borg\bin\borg\archiver.py", line 666, in _process self._process(archive, cache, matcher, exclude_caches, exclude_if_present, NameError: name 'archive' is not defined

[Anakonda]

The build script uses modulefinder to collect all modules needed to run borg. For some reason it does not always find all required modules. Then you get import errors. I have tried to manually add modules it misses on appveyor.

[Unkn0wn-MDCLXIV]

I tried, but there are a lot of other problems with the current code as well. It tries to execute os.statvfs, which does not exist in windows. And even after fixing that, it still crashes with errors I wasnt able to fix.

Whats wrong with the prebuilt binary here tho? Why does it say archive is not defined? Maybe we can work on fixing that, so we have at least one natively working windows binary.

[Anakonda]

Can you test this branch: https://github.com/Anakonda/borg/tree/windows_fixes It should fix archive not being defined.

If you find any other errors I will try fix them and then do a PR with all of them.

[Unkn0wn-MDCLXIV]

That one is working!

However, I get this warning when starting the backup: Warning: permissions to read auditing settings (SACL) denied. Try running as admin.

And I can see no stats while backing up. It stays at 0 B O 0 B C 0 B D 0 N

[Unkn0wn-MDCLXIV]

@Anakonda Is there a reason why the folder structure is messed up when backing up with windows? I have my borg repo mounted, and when I browse through the backup created with windows there is no folder structure or clearnames at all, just thousands of files with random names.

[Anakonda]

Native folder separator is used so foo\bar is written as the file name on windows. Extracting on non-windows it does not have the same meaning as / is used as folder separator

[Unkn0wn-MDCLXIV]

Oh yeah, that makes sense. Any way we can work arround that?

[Anakonda]

Need to use / as separator in windows too.

[Unkn0wn-MDCLXIV]

Line 471 - 472 in archiver.py says if sys.platform == 'win32': path = posixpath.normpath(path.replace('\\', '/'))

Why doesnt that do the job?

[henfri]

Hello, Where do I find the most recent windows binary? Greetings, Hendrik

[Anakonda]

https://ci.appveyor.com/project/borgbackup/borg/build/artifacts

[quicktrick]

Hi guys! Could you please give me some points on how to use this port on Windows 10? I extracted windowsbuild.zip to some folder, extracted library.zip too, launched a command prompt as Administrator, but when I try to start the program I get:

d:\borgbackup>borg help
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000a3c (most recent call first):

d:\borgbackup>

What am I doing wrong?

[Anakonda]

Do not extract 'library.zip' Those are automated builds and may be broken.

[quicktrick]

Thanks for your reply, @Anakonda With not extracted library.zip the output is:

D:\borgbackup>borg help
Failed to import the site module
ModuleNotFoundError: No module named 'site'

D:\borgbackup>

[enkore]

If you're looking for a Windows backup software for productive use, it's probably a better idea to stick to something better tested on Windows.

[quicktrick]

You are probably right @enkore. I'm usually using another backup solutions on Windows. And I'm also using Borg backup on about 30 Linux and FreeBSD servers. Thanks for your reply anyway. There are some mixed solutions of Borg backup on Linux + Windows I'm going to consider.

[henfri]

Hello,

sorry, but where under https://ci.appveyor.com/project/borgbackup/borg/build/artifacts do I find the download?

Appart from that: What's the future of the windows-builds. I find it a pitty, that this work is not merged into the dev-branch. Is that even still possible? Or would it be easier to re-start the modifications from the current dev-branch?

Regards, Hendrik

[henfri]

Hello -again,

apart from my previous question: I still have an older executable (1.1) for windows. I run

pageant my_key.ppk
plink borg@myserver

--> I can login without password. But If I run borg now, I am asked for a bassword. What could be the reason for that?

Regards, Hendrik

[ThomasWaldmann]

@henfri can't tell much about the windows stuff except that it needs more developers, reviewers, testers and a more continued effort. I can only review very simple windows related changed, to review more advanced stuff it needs developers with experience of windows fs and apis.

[henfri]

Hello, this is a bit of a hen and egg problem: I am sure I can test and fix -but only from something that is running. Also, I am concerned about the big gap between dev and the windows branch. For me, it seems neccessary to first integrate the existing modifications into the master branch to avoid the branches diverging.

Greetings, Hendrik

[ThomasWaldmann]

There is more recent windows related work in master branch that was done independently of the windows branch. It was smaller, limited scope stuff that I could review and so it got merged. Maybe doing the smaller, simpler steps first makes sense.

[henfri]

yes. Do you know, whether the current master branch works in windows?

I think, that also we can distinguish between: a) running in windows in a python distribution b) a 'compiled' (frozen) executable

In order not to loose the existing work though, it would be good if someone familiar with it could review the old commits in the windows branch, adopt them into the current master and create (small) pull requests.