bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.21k stars 4.07k forks source link

tags=["exclusive-if-local"] limits concurrency when running remotely #17834

Open EdSchouten opened 1 year ago

EdSchouten commented 1 year ago

Description of the bug:

Commit 8936828610db8106864e41860ad86c5c415fa0ed added tags=["exclusive-if-local"], which can be used to only limit the concurrency of a test, if and only if it runs locally. When running remotely, it should be permitted to run at full concurrency.

Unfortunately, it looks like only one default implementation of forceExclusiveIfLocalTestsInParallel() is provided, and that one always returns false. This means that if remote execution is used using --remote_executor, it will also limit concurrency. This is obviously not intended.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

WORKSPACE

register_execution_platforms("//:linux_x86_64")

BUILD

platform(
    name = "linux_x86_64",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
    remote_execution_properties = """
        Your platform properties go here
        """,
)

sh_test(
    name = "foo_test",
    srcs = ["foo_test.sh"],
    #tags = ["exclusive-if-local"],
)

foo_test.sh

#!/bin/sh
echo it worked

Then run a command like this:

bazel test --remote_executor=... --remote_instance_name=... :foo_test --runs_per_test 10000

On my laptop this will run 16 tests in parallel. But if I uncomment the tags = argument, it limits it to just 1 test.

Which operating system are you running Bazel on?

macOS Ventura

What is the output of bazel info release?

release 6.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

fmeum commented 1 year ago

@susinmotion Is it possible that exclusive-if-local is only honored by Google's remote executor, but not the OSS implementation of the RE protocol?

github-actions[bot] commented 5 months ago

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

p0deje commented 5 months ago

It's still relevant and important for execution in RBE environments.

nlopezgi commented 3 months ago

This is still causing issues for RE users. It would be awesome if the fix could be merged.