bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.23k stars 4.07k forks source link

Windows py_test data glob exclude_directories = 0 bug #3674

Closed aj-michael closed 5 years ago

aj-michael commented 7 years ago

With Bazel 0.5.4, the following test passes on Linux but fails to build on Windows. This is blocking #3663, because our Linux Android integration tests use a glob with exclude_directories = 0 to pull in the entire Android SDK and NDK.

$ cat BUILD
py_test(
    name = "test",
    srcs = ["test.py"],
    data = glob(["some_directory"], exclude_directories = 0),
)

$ cat test.py
# Nothing

$ file some_directory/
some_directory/: directory

$ bazel test //:test -s
____Loading complete.  Analyzing...
____Found 1 test target...
____Building...
____[0 / 4] PythonZipper test.zip
>>>>>>>>> # //:test [action 'PythonZipper test.zip']
cd C:/msys64/tmp/_bazel_a/agvwodbr/execroot/io_bazel
  SET PATH=C:\msys64\usr\bin;C:\msys64\bin;C:\msys64\usr\local\bin;C:\msys64\usr\bin;C:\msys64\usr\bin;C:\msys64\opt\bin;C:\Windows\System32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
  external/bazel_tools/tools/zip/zipper/zipper.exe cC bazel-out/msvc_x64-fastbuild/bin/test.zip @bazel-out/msvc_x64-fastbuild/bin/test.zip-2.params
ERROR: C:/users/a/programming/bazel/BUILD:113:1: PythonZipper test.zip failed (Exit -1).
File some_directory does not seem to exist.Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
____Elapsed time: 0.993s, Critical Path: 0.02s

Executed 0 out of 1 test: 1 fails to build.
aj-michael commented 7 years ago

For context, I want to add an android integration test under //src/test/py/bazel that depends on this filegroup: https://github.com/bazelbuild/bazel/blob/663df9d96f10a2dd0fa8a85b4aac39cd2cc368fb/tools/android/android_sdk_repository_template.bzl#L57-L69

meteorcloudy commented 7 years ago

The failure is because zipper.exe couldn't take an directory path as argument, it will consider it as a file and throw the error File some_directory does not seem to exist.

Can you use glob(["some_directory/**"]), to work around this?

aj-michael commented 7 years ago

I can try that tomorrow, however I think the reason that we don't do that is because the glob is very large and would take a long time if we use **. Perhaps @ahumesky has some background on that.

aj-michael commented 7 years ago

Unfortunately, the filegroup that I'd like to depend on is baked into the Bazel binary. So I can use the workaround, however, I will have to wait for another Bazel release before being able to use it in the test I'm writing.

aj-michael commented 7 years ago

BTW, this is not specific to globs. It happens whenever there is a directory in the runfiles of the test. E.g. a filegroup with srcs = ['some_directory'].

aj-michael commented 7 years ago

I see that the failing action is triggered by the --build_python_zip flag which defaults to true for Windows but False for other platforms. However, --build_python_zip=0 causes a bazel crash on Windows.

$ bazel test ///:test -s --build_python_zip=no
____Loading package:
____Loading package: @bazel_tools//tools/cpp
____Loading package: @local_config_xcode//
____Loading package: @local_config_cc//
____Loading package: @local_jdk//
____Loading complete.  Analyzing...
____Loading package: tools/defaults
____Loading package: @bazel_tools//tools/python
____Loading package: @bazel_tools//tools/launcher
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:test 38887c834cd6ab8e89407cb74612183b (1170565475 82654580)' (requested by nodes )
____Elapsed time: 2.696s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:test 38887c834cd6ab8e89407cb74612183b (1170565475 82654580)' (requested by nodes )
        at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:475)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: py_test rule //:test : These artifacts do not have a generating action:
bazel-out/msvc_x64-fastbuild/bin/test.cmd
No origin, run with --experimental_extended_sanity_checks
These actions were checked:
com.google.devtools.build.lib.analysis.actions.TemplateExpansionAction TemplateExpand
    bazel-out/msvc_x64-fastbuild/bin/test
com.google.devtools.build.lib.actions.MiddlemanAction Middleman
    bazel-out/msvc_x64-fastbuild/internal/_middlemen/test-runfiles_artifacts
com.google.devtools.build.lib.analysis.SourceManifestAction SourceSymlinkManifest
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles_manifest
com.google.devtools.build.lib.analysis.actions.SymlinkTreeAction SymlinkTree
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles/MANIFEST
com.google.devtools.build.lib.actions.MiddlemanAction Middleman
    bazel-out/msvc_x64-fastbuild/internal/_middlemen/test-runfiles
com.google.devtools.build.lib.analysis.SourceManifestAction PackagingSourcesManifest
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles.SOURCES
com.google.devtools.build.lib.rules.test.TestRunnerAction TestRunner
    bazel-out/msvc_x64-fastbuild/testlogs/test/test.log
    bazel-out/msvc_x64-fastbuild/testlogs/test/test.cache_status

        at com.google.devtools.build.lib.analysis.CachingAnalysisEnvironment.verifyGeneratedArtifactHaveActions(CachingAnalysisEnvironment.java:150)
        at com.google.devtools.build.lib.analysis.CachingAnalysisEnvironment.disable(CachingAnalysisEnvironment.java:113)
        at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:1185)
        at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:269)
        at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:400)
        ... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'CONFIGURED_TARGET://:test 38887c834cd6ab8e89407cb74612183b (1170565475 82654580)' (requested by nodes )
        at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:475)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:352)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: py_test rule //:test : These artifacts do not have a generating action:
bazel-out/msvc_x64-fastbuild/bin/test.cmd
No origin, run with --experimental_extended_sanity_checks
These actions were checked:
com.google.devtools.build.lib.analysis.actions.TemplateExpansionAction TemplateExpand
    bazel-out/msvc_x64-fastbuild/bin/test
com.google.devtools.build.lib.actions.MiddlemanAction Middleman
    bazel-out/msvc_x64-fastbuild/internal/_middlemen/test-runfiles_artifacts
com.google.devtools.build.lib.analysis.SourceManifestAction SourceSymlinkManifest
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles_manifest
com.google.devtools.build.lib.analysis.actions.SymlinkTreeAction SymlinkTree
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles/MANIFEST
com.google.devtools.build.lib.actions.MiddlemanAction Middleman
    bazel-out/msvc_x64-fastbuild/internal/_middlemen/test-runfiles
com.google.devtools.build.lib.analysis.SourceManifestAction PackagingSourcesManifest
    bazel-out/msvc_x64-fastbuild/bin/test.runfiles.SOURCES
com.google.devtools.build.lib.rules.test.TestRunnerAction TestRunner
    bazel-out/msvc_x64-fastbuild/testlogs/test/test.log
    bazel-out/msvc_x64-fastbuild/testlogs/test/test.cache_status

        at com.google.devtools.build.lib.analysis.CachingAnalysisEnvironment.verifyGeneratedArtifactHaveActions(CachingAnalysisEnvironment.java:150)
        at com.google.devtools.build.lib.analysis.CachingAnalysisEnvironment.disable(CachingAnalysisEnvironment.java:113)
        at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.createConfiguredTarget(ConfiguredTargetFunction.java:1185)
        at com.google.devtools.build.lib.skyframe.ConfiguredTargetFunction.compute(ConfiguredTargetFunction.java:269)
        at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:400)
        ... 4 more
laszlocsomor commented 5 years ago

This works with Bazel 0.23.1