bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.94k stars 4.02k forks source link

Expose workspace provenance for strategy selection #18244

Open jmmv opened 1 year ago

jmmv commented 1 year ago

Description of the feature request:

Bazel has the --strategy and --regexp_strategy flags to customize how certain actions are run. Unfortunately, when importing an external Bazel build via the WORKSPACE, it is impossible to apply different strategies to the actions that come from the external build. It'd be nice if there was some way of knowing this. As an idea, maybe the action descriptions could unconditionally include the name of the external workspace they came from so that this information could be used in combination with --regexp_strategy.

What underlying problem are you trying to solve with this feature?

We have a project with significant code of our own that relies on a bunch of third-party dependencies.

The third-party dependencies rarely change so they can be reused from the remote cache the vast majority of the time, and we want to make sure these are reused from the cache.

Our code, however, generates large artifacts (think large C++ binaries), and ferrying them back and forth from the remote build farm is really costly and slow—much more than just letting those actions run locally—so we want to keep these building locally for the benefit of quick iterative incremental builds.

We want to configure Bazel to make it forcibly run the build of the external dependencies on the build farm, and build our "internal" dependencies either locally or with the dynamic strategy. So far we have been able to do this because all of our code is in subdirectories with rather unique names, so --strategy_regexp can be used to special-case those actions... but this is very fragile. There is no guarantee that the action descriptions contain the path name (thus this matching could fail), and there is no guarantee that our relative paths will remain "unique".

It'd be best if Bazel natively exposed the workspace each action came from so that we could apply this type of filtering.

Which operating system are you running Bazel on?

N/A

What is the output of bazel info release?

bazel-6.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

larsrc-google commented 1 year ago

Would having remote-cache as a strategy be enough to fix this?

github-actions[bot] commented 2 months ago

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.