search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.97k stars 4.03k forks source link

Remote coverage does not have a configuration #19781

Open stevebarrau opened 11 months ago

stevebarrau commented 11 months ago

Description of the bug:

I am trying to get coverage to work in our RBE cluster.

Collecting the coverage works locally with:

$ bazel coverage -s --combined_report=lcov //python: trivial_test
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
INFO: LCOV coverage report is located at /private/var/tmp/_bazel_steve/988dbe4dd5ee602e299c1f37c1a3d6b2/execroot/federation_example/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
INFO: From Coverage report generation:
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: Found 1 tracefiles.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main parseFilesSequentially
INFO: Parsing file bazel-out/darwin-fastbuild/testlogs/python/trivial_test/coverage.dat
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovJsonInfoFiles
INFO: No gcov json file found.
Oct 10, 2023 12:09:19 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Target //python: trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 45.101s, Critical Path: 4.29s
INFO: 15 processes: 4 disk cache hit, 10 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 15 total actions
//python: trivial_test                                            (cached) PASSED in 4.0s
  XXX/execroot/federation_example/bazel-out/darwin-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 0 out of 1 test: 1 test passes.

Running the same with RBE:

$ bazel-with-our-rbe coverage --combined_report=lcov  --experimental_fetch_all_coverage_outputs --experimental_split_coverage_postprocessing -s //python:trivial_test
....
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/k8-opt-exec-BAC2A3E1/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
ERROR: <builtin>: Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:244)
    at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1499)
    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:269)
    at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:244)
    at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:301)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:152)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:112)
    at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:46)
    at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:152)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:961)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1128)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1086)
    at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:160)
    at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:93)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:519)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:827)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:323)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:161)
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:562)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:365)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
    at io.grpc.Status.asRuntimeException(Status.java:535)
    at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:648)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:178)
    at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:149)
    at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:523)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:147)
    ... 28 more
Target //python:trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 1.548s, Critical Path: 0.67s
INFO: 3 processes: 1 remote cache hit, 2 internal.
FAILED: Build did NOT complete successfully
//python: trivial_test                                            (cached) PASSED in 1.0s
  XXX/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 0 out of 1 test: 1 test passes.
All tests passed but there were other errors during the build.

I am puzzled by the Configuration: system bit. IIUC the CoverageReportAction should spawn a BasicSpawn with a configuration. In our RBE implementation this means we need to manually specify the exec properties using --remote_default_platform_properties to get this to work remotely. We would like to avoid using remote_default_platform_properties as a workaround given this probably creates situations where if a spawn does not have a platform set, it will run them incorrectly.

Is this the expected behavior or is there an oversight is setting the configuration for the CoverageReportAction? It feels like either the BasicSpawn needs to inherit a platform (e.g. from the host platform or the default exec platform), or there needs to be a way to specify the coverage action's flag (e.g. a --coverage_platform flat or similar)?

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

# python/trivial.py
def foo():
    return 1
# python/trivial_test.py
import unittest

from python import trivial

class TrivialTest(unittest.TestCase):
    def test_trivial(self):
        self.assertEqual(1, trivial.foo())

Use rules_python and a wrapper for unittest to get XML outputs.

Which operating system are you running Bazel on?

macOS 12.6.3

What is the output of bazel info release?

release 6.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

I looked for coverage configuration to no avail.

Any other information, logs, or outputs that you want to share?

No response

illicitonion commented 11 months ago

@coeuvre and @fmeum - I think you've been working at the intersection of coverage + remote execution(e.g. https://github.com/bazelbuild/bazel/pull/16556) - any thoughts on how the merger action should be getting a platform?

fmeum commented 11 months ago

Could you check whether https://github.com/bazelbuild/bazel/pull/19784 fixes this? It's pretty hacky though and untested (Bazel CI passes with it). It also requires Skymeld.

stevebarrau commented 11 months ago

I checked on 7b28c63f02287a32ab05ac21d9e0cc9be53f80f7.

The Configuration is still "system" but now our RBE platform is getting picked up in the Execution platform. We looked into how execProperties are set based on the platform and found a code path where only remoteExecutionProperties are used, and not execProperties. I filed a separate PR with the fix here: https://github.com/bazelbuild/bazel/pull/19792

This separate PR + the changes in this PR makes coverage pass in our use case.

$ bazel-with-our-rbe coverage -s --combined_report=lcov //python:trivial_test
...
SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: CoverageReport]
(cd XXX/execroot/federation_example && \
  exec env - \
    JAVA_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
    PYTHON_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \
  bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat')
# Configuration: system
# Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64
ERROR: <builtin>: Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241)
    at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1490)
    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:286)
    at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:259)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:156)
    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:116)
    at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
    at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:140)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1135)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1052)
    at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
    at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:553)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:852)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:331)
    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:169)
    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
    at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {}
    at io.grpc.Status.asRuntimeException(Status.java:535)
    at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175)
    at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
    at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146)
    at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:526)
    at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144)
    ... 26 more
Target //python:trivial_test up-to-date:
  bazel-bin/python/trivial_test
INFO: Elapsed time: 248.538s, Critical Path: 119.85s
INFO: 15 processes: 3 remote cache hit, 11 internal, 1 remote.
ERROR: Build did NOT complete successfully
//python:trivial_test                                                     PASSED in 1.4s
  XXX/execroot/federation_example/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat

Executed 1 out of 1 test: 1 test passes.
All tests passed but there were other errors during the build.
Full log ```console $ cat .bazelrc | rg coverage coverage:remote --experimental_fetch_all_coverage_outputs coverage:remote --experimental_split_coverage_postprocessing $ bazel-with-our-rbe info release development version $ bazel-with-our-rbe coverage -s --combined_report=lcov //python:trivial_test INFO: Invocation ID: 0a2f4994-9ef5-498a-a4fd-4ef313cd9bbf INFO: Using default value for --instrumentation_filter: "^//python[/:]". INFO: Override the above default with --instrumentation_filter INFO: Analyzed target //python:trivial_test (134 packages loaded, 14289 targets configured). SUBCOMMAND: # @rules_java_builtin//toolchains:platformclasspath [action 'JavaToolchainCompileClasses external/rules_java_builtin/toolchains/platformclasspath_classes [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: JavaToolchainCompileClasses] (cd XXX/execroot/federation_example && \ exec env - \ external/jdk_17_linux/bin/javac -source 8 -target 8 -Xlint:-options -d bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath_classes external/rules_java_builtin/toolchains/DumpPlatformClassPath.java) # Configuration: 09XXX61 # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 SUBCOMMAND: # @rules_java_builtin//toolchains:platformclasspath [action 'JavaToolchainCompileBootClasspath external/rules_java_builtin/toolchains/platformclasspath.jar [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: JavaToolchainCompileBootClasspath] (cd XXX/execroot/federation_example && \ exec env - \ external/jdk_17_linux/bin/java -XX:+IgnoreUnrecognizedVMOptions '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.platform=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' -cp bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath_classes DumpPlatformClassPath bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/rules_java_builtin/toolchains/platformclasspath.jar external/jdk_17_linux) # Configuration: 09XXX61 # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 SUBCOMMAND: # @remote_coverage_tools//:Main [action 'Building external/remote_coverage_tools/Main.jar () [for tool]', configuration: 09XXX61, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: Javac] (cd XXX/execroot/federation_example && \ exec env - \ LC_CTYPE=en_US.UTF-8 \ PATH=/bin:/usr/bin:/usr/local/bin \ external/jdk_17_linux/bin/java '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.lang=ALL-UNNAMED' '-Dsun.io.useCanonCaches=false' -XX:-CompactStrings -Xlog:disable '-Xlog:all=warning:stderr:uptime,level,tags' -jar external/remote_java_tools/java_tools/JavaBuilder_deploy.jar @bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.jar-0.params @bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.jar-1.params) # Configuration: 09XXX61 # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 SUBCOMMAND: # //python:trivial_test [action 'Testing //python:trivial_test', configuration: a3XXX79a, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: TestRunner] (cd XXX/execroot/federation_example && \ exec env - \ BAZEL_CC_COVERAGE_TOOL=GCOV \ BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \ CC_CODE_COVERAGE_SCRIPT=external/bazel_tools/tools/test/collect_cc_coverage.sh \ COVERAGE_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/_coverage \ COVERAGE_MANIFEST=bazel-out/k8-fastbuild/bin/python/trivial_test.instrumented_files \ COVERAGE_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat \ EXPERIMENTAL_SPLIT_XML_GENERATION=1 \ IS_COVERAGE_SPAWN=0 \ JAVA_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ LCOV_MERGER=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main \ PATH=/bin:/usr/bin:/usr/sbin \ PYTHON_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ RUNFILES_DIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ RUNTEST_PRESERVE_CWD=1 \ RUN_UNDER_RUNFILES=1 \ SPLIT_COVERAGE_POST_PROCESSING=1 \ TEST_BINARY=python/trivial_test \ TEST_INFRASTRUCTURE_FAILURE_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.infrastructure_failure \ TEST_LOGSPLITTER_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.raw_splitlogs/test.splitlogs \ TEST_PREMATURE_EXIT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.exited_prematurely \ TEST_SIZE=small \ TEST_SRCDIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ TEST_TARGET=//python:trivial_test \ TEST_TIMEOUT=300 \ TEST_TMPDIR=_tmp/ef05b1d4578920cbc6e63775f3dc0d1d \ TEST_UNDECLARED_OUTPUTS_ANNOTATIONS=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/ANNOTATIONS \ TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest \ TEST_UNDECLARED_OUTPUTS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs \ TEST_UNDECLARED_OUTPUTS_MANIFEST=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/MANIFEST \ TEST_UNDECLARED_OUTPUTS_ZIP=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs/outputs.zip \ TEST_UNUSED_RUNFILES_LOG_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.unused_runfiles_log \ TEST_WARNINGS_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.warnings \ TEST_WORKSPACE=federation_example \ TZ=UTC \ XML_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.xml \ external/bazel_tools/tools/test/test-setup.sh external/bazel_tools/tools/test/collect_coverage.sh python/trivial_test) # Configuration: a3XXX79a # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 SUBCOMMAND: # //python:trivial_test [action 'Testing //python:trivial_test', configuration: a3XXX79a, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: TestRunner] (cd XXX/execroot/federation_example && \ exec env - \ BAZEL_CC_COVERAGE_TOOL=GCOV \ BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \ CC_CODE_COVERAGE_SCRIPT=external/bazel_tools/tools/test/collect_cc_coverage.sh \ COVERAGE_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/_coverage \ COVERAGE_MANIFEST=bazel-out/k8-fastbuild/bin/python/trivial_test.instrumented_files \ COVERAGE_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat \ EXPERIMENTAL_SPLIT_XML_GENERATION=1 \ IS_COVERAGE_SPAWN=1 \ JAVA_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ LCOV_MERGER=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main \ PATH=/bin:/usr/bin:/usr/sbin \ PYTHON_RUNFILES=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ RUNFILES_DIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ RUNTEST_PRESERVE_CWD=1 \ RUN_UNDER_RUNFILES=1 \ SPLIT_COVERAGE_POST_PROCESSING=1 \ TEST_BINARY=python/trivial_test \ TEST_INFRASTRUCTURE_FAILURE_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.infrastructure_failure \ TEST_LOGSPLITTER_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.raw_splitlogs/test.splitlogs \ TEST_NAME=//python:trivial_test \ TEST_PREMATURE_EXIT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.exited_prematurely \ TEST_SHARD_INDEX=0 \ TEST_SIZE=small \ TEST_SRCDIR=bazel-out/k8-fastbuild/bin/python/trivial_test.runfiles \ TEST_TARGET=//python:trivial_test \ TEST_TIMEOUT=300 \ TEST_TMPDIR=_tmp/ef05b1d4578920cbc6e63775f3dc0d1d \ TEST_TOTAL_SHARDS=0 \ TEST_UNDECLARED_OUTPUTS_ANNOTATIONS=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/ANNOTATIONS \ TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest \ TEST_UNDECLARED_OUTPUTS_DIR=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs \ TEST_UNDECLARED_OUTPUTS_MANIFEST=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs_manifest/MANIFEST \ TEST_UNDECLARED_OUTPUTS_ZIP=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.outputs/outputs.zip \ TEST_UNUSED_RUNFILES_LOG_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.unused_runfiles_log \ TEST_WARNINGS_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.warnings \ TEST_WORKSPACE=federation_example \ TZ=UTC \ XML_OUTPUT_FILE=bazel-out/k8-fastbuild/testlogs/python/trivial_test/test.xml \ external/bazel_tools/tools/test/collect_coverage.sh) # Configuration: a3XXX79a # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 SUBCOMMAND: # (unknown) [action 'Coverage report generation', configuration: system, execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64, mnemonic: CoverageReport] (cd XXX/execroot/federation_example && \ exec env - \ JAVA_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \ PYTHON_RUNFILES=bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main.runfiles \ bazel-out/k8-opt-exec-ST-e4625b82c993/bin/external/remote_coverage_tools/Main '--reports_file=bazel-out/_coverage/lcov_files.tmp' '--output_file=bazel-out/_coverage/_coverage_report.dat') # Configuration: system # Execution platform: @our_rbe_repo//common/platforms:ubi9-x86_64 ERROR: : Coverage report generation failed: (Exit 34): FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {} java.io.IOException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {} at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241) at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1490) at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:286) at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245) at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127) at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116) at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:259) at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:156) at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:116) at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45) at com.google.devtools.build.lib.bazel.coverage.CoverageReportActionBuilder$CoverageReportAction.execute(CoverageReportActionBuilder.java:140) at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1135) at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1052) at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165) at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94) at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:553) at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:852) at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:331) at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:169) at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461) at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414) at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: No workers exist for instance name prefix "" platform {} at io.grpc.Status.asRuntimeException(Status.java:535) at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660) at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175) at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245) at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127) at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116) at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146) at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:526) at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144) ... 26 more Target //python:trivial_test up-to-date: bazel-bin/python/trivial_test INFO: Elapsed time: 248.538s, Critical Path: 119.85s INFO: 15 processes: 3 remote cache hit, 11 internal, 1 remote. ERROR: Build did NOT complete successfully //python:trivial_test PASSED in 1.4s XXX/execroot/federation_example/bazel-out/k8-fastbuild/testlogs/python/trivial_test/coverage.dat Executed 1 out of 1 test: 1 test passes. All tests passed but there were other errors during the build. ```
stevebarrau commented 11 months ago

Note: with this PR; setting remote_execution_properties causes the following NPE:

$ bazel-with-our-rbe -s --combined_report=lcov //python:trivial_test
Running host JVM under debugger (listening on TCP port 5005).
Starting local Bazel server and connecting to it...
... still trying to connect to local Bazel server (72354) after 10 seconds ...
INFO: Invocation ID: ee7ac809-1ca1-4904-82d4-e767829cb641
INFO: Using default value for --instrumentation_filter: "^//python[/:]".
INFO: Override the above default with --instrumentation_filter
ERROR: XXX/external/our_rbe/common/platforms/BUILD.bazel:40:9: in exec_properties attribute of platform rule @our_rbe//common/platforms:ubi9-x86_64: Platform contains both remote_execution_properties and exec_properties. Prefer exec_properties over the deprecated remote_execution_properties.
ERROR: XXX/external/our_rbe/common/platforms/BUILD.bazel:40:9: Analysis of target '@our_rbe//common/platforms:ubi9-x86_64' failed
Analyzing: target //python: trivial_test (7 packages loaded, 12 targets configured)
[1 / 1] checking cached actions
FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.NullPointerException
    at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903)
    at com.google.devtools.build.lib.skyframe.BuildResultListener.getHostPlatformInfo(BuildResultListener.java:122)
    at com.google.devtools.build.lib.skyframe.SkyframeBuildView.analyzeAndExecuteTargets(SkyframeBuildView.java:732)
    at com.google.devtools.build.lib.analysis.BuildView.update(BuildView.java:293)
    at com.google.devtools.build.lib.buildtool.AnalysisAndExecutionPhaseRunner.runAnalysisAndExecutionPhase(AnalysisAndExecutionPhaseRunner.java:241)
    at com.google.devtools.build.lib.buildtool.AnalysisAndExecutionPhaseRunner.execute(AnalysisAndExecutionPhaseRunner.java:139)
    at com.google.devtools.build.lib.buildtool.BuildTool.buildTargetsWithMergedAnalysisExecution(BuildTool.java:305)
    at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:173)
    at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:510)
    at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:478)
    at com.google.devtools.build.lib.runtime.commands.TestCommand.doTest(TestCommand.java:163)
    at com.google.devtools.build.lib.runtime.commands.TestCommand.exec(TestCommand.java:116)
    at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:664)
    at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:244)
    at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)
    at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:621)
    at io.grpc.Context$1.run(Context.java:566)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
fmeum commented 11 months ago

I can look into cleaning up the PR after yours has been merged, but I may not get to it this month.

katre commented 10 months ago

This is not configurability.