Windows support

[mattmoor]

Opening this as a general tracking issue for PAR support for Bazel on Windows.

The first issue I'm hitting seems to be where the PAR compiler parses the stub file.

[duggelz]

I investigated, and neither TravisCI nor CircleCI have Windows support. So I may try Kokoro, which has the downside that it's only accessible to Googlers.

[mattmoor]

@damienmg is it possible to add this to Bazel CI?

[damienmg]

windows-x86_64 is the node name to select a windows slave. You can also just add it to your PR and iterate over your PR until the windows port works well.

[damienmg]

And yes we can add google/subpar to our ci, see https://github.com/bazelbuild/continuous-integration/blob/master/docs/owner.md for the documentation (basically on our CI we add it and you have to add @bazel-io to have write access to the repository, the rest is just fiddling the configuration and can be done on a PR).

[laszlocsomor]

For whomever will work on this issue: if making subpar work on Windows includes dealing with long paths, be aware that we (the Bazel team) learned a lot about Windows path handling as we ported Bazel to Windows, and I'd be happy to share my knowledge regarding \\?\ and the caveats around it.

[j143-zz]

While building a project on windows I received this error: Hope this can be of some help. Any thoughts how to avoid this error ?

D:/repo/my-example-project> bazel build -c opt single_task:tune.par
INFO: Analysed target //single_task:tune.par (0 packages loaded).
INFO: Found 1 target...
ERROR: C:/users/anonymous/appdata/local/temp/_bazel_vikalpa/ykleqatt/external/subpar/compiler/BUILD:40:1: Building par file @subpar//compiler:compiler.par failed (Exit 1)
Traceback (most recent call last):
  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmpc57_f4df815e42ecaaeb\Bazel.runfiles_meskn2o8\runfiles\subpar\compiler\compi
ler.py", line 23, in <module>
    sys.exit(cli.main(sys.argv))
  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmpc57_f4df815e42ecaaeb\Bazel.runfiles_meskn2o8\runfiles\subpar\compiler\cli.p
y", line 104, in main
    import_roots, interpreter = parse_stub(args.stub_file)
  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmpc57_f4df815e42ecaaeb\Bazel.runfiles_meskn2o8\runfiles\subpar\compiler\cli.p
y", line 71, in parse_stub
    for line in stub_file:
  File "C:\Users\anonymous\AppData\Local\Programs\Python\Python36\lib\codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
Target //single_task:tune.par failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.086s, Critical Path: 0.46s
FAILED: Build did NOT complete successfully

Thank you.

[j143-zz]

Hi @laszlocsomor - can you share how resolve long path issues.

[laszlocsomor]

Hi @j143 , I don't think this error is related to long paths. Python3 always prints Windows paths in stack traces with the \\?\ prefix.

@mattmoor 's in his original post was referring to https://github.com/google/subpar/blob/a32c2783d2b882b064123de942572f2e69451610/compiler/cli.py#L55, and your stack trace references the same function (but not the same version of the file, apparently).

The line numbers in your stack trace do not line up with the current version of compiler/cli.py, but they do line up with this earlier version.

From the error message, this looks like an encoding error. It might be fixed in the latest version of the file. To try that, you should use a newer version of subpar. And I believe you need to update your WORKSPACE file for that.

Could you show me the definition of the subpar rule in your WORKSPACE file? Either copy-paste it here, or run bazel query --output=build //external:subpar and paste its output here.

[j143-zz]

cc @laszlocsomor

1. WORKSPACE contains

   git_repository(
   name = "subpar",
   remote = "https://github.com/google/subpar",
   tag = "1.0.0",
   )

   2.with the command

   > bazel query single_task:tune.par
   ........................................................
   //single_task:tune.par

[laszlocsomor]

In the WORKSPACE file, try setting the tag to 1.3.0, that is the latest release.

I don't know if that will help though (I don't work on subpar), but it's worth trying.

[j143-zz]

Thank you @laszlocsomor . I am pasting the log file, so somebody might help. It is the same error: with tag 1.3.0

  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmp70a_a377239f5a145296\Bazel.runfiles_379_n674\runfiles\subpar\compiler\compiler.py", line 23, in <module>
    sys.exit(cli.main(sys.argv))
  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmp70a_a377239f5a145296\Bazel.runfiles_379_n674\runfiles\subpar\compiler\cli.py", line 139, in main
    import_roots, interpreter = parse_stub(args.stub_file)
  File "\\?\C:\users\anonymous\appdata\local\temp\_bazel_vikalpa\ykleqatt\execroot\__main__\tmp70a_a377239f5a145296\Bazel.runfiles_379_n674\runfiles\subpar\compiler\cli.py", line 106, in parse_stub
    for line in stub_file:
  File "C:\Users\anonymous\AppData\Local\Programs\Python\Python36\lib\codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

[UebelAndre]

Does anyone know if there's been any progress here?

[shridevinb]

Does anyone have an update on this ? any workaround?

[bshashank]

I spent some time on this last week and I want to share my findings.

The root of the issue - as I understand it, is that the built PAR file can never be run directly on Windows.

i.e, you can't CreateProcessW('C:\path\to\foo.par', 'arg1', '...'). Windows will just complain that the par file is not a binary.

So the next best thing is to invoke it as: CreateProcessW('C:\path\to\python.exe', ['C:\path\to\foo.par', 'arg1', ...])

And that's the change we need to make in the starlark rule that invokes the compiler: https://github.com/google/subpar/blob/master/subpar.bzl#L94

Of course, the version of python we use should be inferred via the py_runtime as defined by the user. That sidesteps the issue of having to parse the stub etc, and will provide a cleaner interface, IMO.

There is one wrinkle to be aware of - in the "bootstrap" phase - i.e when you are building the compiler.par. Here we are actually running a py_binary of the compiler.py - which is just the compiler.exe. During this case - we'll want to not prefix the python.exe.

(This has some overlap with PR https://github.com/google/subpar/pull/128)