search

bazelbuild / rules_proto

Protocol buffer rules for Bazel
Apache License 2.0
163 stars 70 forks source link

Windows cc protobuf building is broken #87

Closed andrewkatson closed 2 years ago

andrewkatson commented 3 years ago

If you try to use compiler=mingw-gcc on Windows you cannot build any protos. 

$ bazel build Security/Proto:all  --javabase=@bazel_tools//tools/jdk:remote_jdk11 --compiler=mingw-gcc
Loading:
Loading: 0 packages loaded
Analyzing: 4 targets (1 packages loaded, 0 targets configured)
INFO: Analyzed 4 targets (1 packages loaded, 6 targets configured).
INFO: Found 4 targets...
[0 / 10] [Prepa] BazelWorkspaceStatusAction stable-status.txt ... (3 actions, 0 running)
ERROR: C:/users/katso/documents/github/denarii/external/keirospublic/Security/Proto/BUILD:15:14: Generating Descriptor Set proto_library //Security/Proto:identifier_proto failed: (Exit -1073741515): protoc.exe failed: error executing command bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe --descriptor_set_out=bazel-out/x64_windows-fastbuild/bin/Security/Proto/identifier_proto-descriptor-set.proto.bin ... (remaining 5 argument(s) skipped)
INFO: Elapsed time: 0.388s, Critical Path: 0.03s
INFO: 5 processes: 5 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

And the error message is not particularly helpful which compounds the issue.

DrewGlinsman commented 3 years ago

Mingw-gcc is not working for me either.

Yannic commented 3 years ago

I don't have a Windows machine to repro this. Would you mind running with --verbose_failures and share the output?

andrewkatson commented 3 years ago 
Loading:
Loading: 0 packages loaded
Analyzing: 14 targets (0 packages loaded, 0 targets configured)
INFO: Analyzed 14 targets (0 packages loaded, 0 targets configured).
INFO: Found 14 targets...
[0 / 36] [Prepa] BazelWorkspaceStatusAction stable-status.txt
ERROR: C:/msys64/home/katso/_bazel_katso/cjrunvaw/external/com_google_protobuf/BUILD:309:15: Generating Java (Immutable) proto_library @com_google_protobuf//:timestamp_proto failed: (Exit -1073741515): protoc.exe failed: error executing command
  cd C:/msys64/home/katso/_bazel_katso/cjrunvaw/execroot/keiros_public
  SET PATH=C:\msys64\usr\bin;C:\msys64\bin;C:\Windows;C:\Windows\System32;C:\Windows\System32\WindowsPowerShell\v1.0;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl;C:\Users\katso\AppData\Local\Microsoft\WindowsApps
    SET RUNFILES_MANIFEST_ONLY=1
  bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe --proto_path=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto --java_out=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/timestamp_proto-speed-src.jar -Igoogle/protobuf/timestamp.proto=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto --direct_dependencies google/protobuf/timestamp.proto --direct_dependencies_violation_msg=%s is imported, but @com_google_protobuf//:timestamp_proto doesn't directly depend on a proto_library that 'srcs' it. bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto
Execution platform: @local_config_platform//:host
ERROR: C:/users/katso/documents/github/denarii/external/keirospublic/Security/Proto/BUILD:64:14 Building Security/Proto/libvalidity_period_proto-speed.jar (1 source jar) failed: (Exit -1073741515): protoc.exe failed: error executing command
  cd C:/msys64/home/katso/_bazel_katso/cjrunvaw/execroot/keiros_public
  SET PATH=C:\msys64\usr\bin;C:\msys64\bin;C:\Windows;C:\Windows\System32;C:\Windows\System32\WindowsPowerShell\v1.0;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\msys64\usr\bin\site_perl;C:\msys64\usr\bin\vendor_perl;C:\msys64\usr\bin\core_perl;C:\Users\katso\AppData\Local\Microsoft\WindowsApps
    SET RUNFILES_MANIFEST_ONLY=1
  bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe --proto_path=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto --java_out=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/timestamp_proto-speed-src.jar -Igoogle/protobuf/timestamp.proto=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto --direct_dependencies google/protobuf/timestamp.proto --direct_dependencies_violation_msg=%s is imported, but @com_google_protobuf//:timestamp_proto doesn't directly depend on a proto_library that 'srcs' it. bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto
Execution platform: @local_config_platform//:host
INFO: Elapsed time: 0.426s, Critical Path: 0.07s
INFO: 9 processes: 9 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
meteorcloudy commented 3 years ago

It looks like the protoc compiler doesn't work when compiled with "mingw-gcc"? You can try to directly run this binary bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe and see what happens?

BTW, why do you want to use compiler=mingw-gcc?

andrewkatson commented 3 years ago

Okay ill try that out. I have some code that for the life of me won't compile with MSVC.

Yeah running that directly works just fine.

meteorcloudy commented 3 years ago

Can you also try the full commmand?

bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe --proto_path=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto --java_out=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/timestamp_proto-speed-src.jar -Igoogle/protobuf/timestamp.proto=bazel-out/x64_windows-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto --direct_dependencies google/protobuf/timestamp.proto
andrewkatson commented 3 years ago

Weird, in my case bazel-out is not a directory but rather a single file. So i cannot run that command. Do you know why this might be?

I got it working in another WORKSPACE. 

C:/Users/katso/Documents/Github/KeirosPublic/bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc.exe: error while loading shared libraries: ?: cannot open shared object file: No such file or directory

Everything I am reading says this means that the path to some shared library is not set in my PATH. But I added the path to the protoc.exe and it still failed. I am guessing it is some other library then?

meteorcloudy commented 3 years ago

@andrewkatson Yes, you can check what libraries it is linked to by ldd <the protoc.exe> (you have run this command in msys)

andrewkatson commented 3 years ago

ahhhh that should be helpful thank you!

        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x7ffcfa410000)
        KERNEL32.DLL => /c/Windows/System32/KERNEL32.DLL (0x7ffcf89f0000)
        KERNELBASE.dll => /c/Windows/System32/KERNELBASE.dll (0x7ffcf7f70000)
        msvcrt.dll => /c/Windows/System32/msvcrt.dll (0x7ffcf86e0000)

It gives me this list. But what is absolutely insane is this is my $PATH

/usr/local/bin:/usr/bin:/bin:/opt/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:C:/Users/katso/Documents/Github/KeirosPublic/bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/:/c/Users/katso/AppData/Local/Microsoft/WindowsApps:/c/Users/katso/Documents/Github/KeirosPublic/bazel-out/x64_windows-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/

You can see that /c/Windows/System32 is already included!!!!

Is this unfixable?

Is it possible it is missing a library which is not showing up?

Interesting. When I use dependency walker instead it says the file uses libwinpthread-1.dll and libstdc++-6.dll

Ah ha!! I ran it in the command shell and it told me which dlls were missing.

Not sure what to do with these though. I tried adding some compiler flags

--copt="-static-libgcc" and --copt="-static-libstdc++" and --copt="-static"

but that did nothing unless I use msys-gcc then those commands work!

But then I get other errors 

[73 / 241] Compiling src/google/protobuf/generated_message_table_driven_lite.cc [for host]; 3s local ... (32 actions, 31 running)
[82 / 241] Compiling src/google/protobuf/generated_message_table_driven_lite.cc [for host]; 4s local ... (32 actions running)
[91 / 241] Compiling src/google/protobuf/generated_message_table_driven_lite.cc [for host]; 5s local ... (32 actions, 31 running)
[103 / 241] Compiling src/google/protobuf/compiler/cpp/cpp_file.cc [for host]; 6s local ... (32 actions running)
[116 / 241] Compiling src/google/protobuf/compiler/cpp/cpp_file.cc [for host]; 7s local ... (32 actions running)
[129 / 241] Compiling src/google/protobuf/text_format.cc [for host]; 4s local ... (32 actions running)
[137 / 241] Compiling src/google/protobuf/text_format.cc [for host]; 6s local ... (32 actions running)
[160 / 241] Compiling src/google/protobuf/descriptor.cc [for host]; 6s local ... (32 actions running)
[172 / 241] Compiling src/google/protobuf/descriptor.cc [for host]; 8s local ... (32 actions, 31 running)
[201 / 241] Compiling src/google/protobuf/descriptor.cc [for host]; 10s local ... (8 actions running)
[211 / 241] Linking external/com_google_protobuf/protoc.exe [for host]; 0s local
ERROR: C:/users/katso/documents/github/keirospublic/Security/Proto/BUILD:16:17: ProtocInvocation Security/Proto/identifier_pb2.py failed: (Exit -1073741515): protoc.exe failed: error executing command
  cd /d C:/msys64/home/katso/_bazel_katso/nz6iu7vj/execroot/keiros_public
bazel-out\host\bin\external\com_google_protobuf\protoc.exe --python_out=bazel-out/x64_windows-fastbuild/bin --proto_path=. --proto_path=bazel-out/x64_windows-fastbuild/bin Security/Proto/identifier.proto
Execution platform: @local_config_platform//:host
INFO: Elapsed time: 21.115s, Critical Path: 14.75s
INFO: 208 processes: 20 internal, 188 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

which doesn't fail when I run it by hand.... another mystery!

So outstanding questions.

  1. why dont those compiler flags work with mingw-gcc but do work with msys-gcc
  2. why does that command fail inside of bazel but not when used outside of it

Based on this https://www.reddit.com/r/Windows10/comments/4o4u4e/file_system_error_1073741515/ and this https://stackoverflow.com/questions/62035689/cmake-linked-library-error-0xc0000135-when-attempting-to-run-on-jenkins-bui it may be that missing the DLLs means that building python proto buf on Windows with msys-gcc is just a nogo.

andrewkatson commented 2 years ago

Okay! MSVC is broken too. The error just gets ignored. I feel like this is kind of important. 

bazel-out\x64_windows-opt-exec-2B5CBBC6\bin\external\com_google_protobuf\protoc.exe --descriptor_set_out=bazel-out/x64_windows-fastbuild/bin/src/centralnode/Proto/critical_values_proto-descriptor-set.proto.bin -Isrc/centralnode/Proto/critical_values.proto=src/centralnode/Proto/critical_values.proto --direct_dependencies src/centralnode/Proto/critical_values.proto --direct_dependencies_violation_msg=%s is imported, but //src/centralnode/Proto:critical_values_proto doesn't directly depend on a proto_library that 'srcs' it. src/centralnode/Proto/critical_values.proto
andrewkatson commented 2 years ago

I have news . If you build in command prompt and set PATH to include 

C:\Windows\SYSTEM32
C:\msys64\usr\bin

it builds. Still curious why the msys terminal fails to build though.

andrewkatson commented 2 years ago
  1. Move the following files into C:\msys64\mingw64\bin ntdll.dll KERNEL32.DLL KERNELBASE.dll msvcrt.dll
  2. bazel clean --expunge
  3. profit (build)

That fixes it!!!!