This repository contains a client implementation of Remote Execution API that works with Remote Execution API SDKs.
Reclient integrates with an existing build system to enable remote execution and caching of build actions.
When used with Server implementation of Remote Execution API, it helps to reduce the build times by applying 2 main techniques:
Most clients are expected to see the performance improvement of their builds after migrating from local to remote builds. However, builds with a high number of deterministic build actions that can be executed in parallel are expected to see the greatest improvement.
Reclient consists of the following main binaries:
This is not an officially supported Google product.
re-client
builds with Bazel. We recommend using
Bazelisk to use the version of Bazel
currently supported by this code base.re-client
also requires gclient
, which can be installed by checking out
depot_tools
and adding the depot_tools
directory to your PATH
.re-client
currently builds and is supported on Linux / Mac / Windows.
Once you've installed Bazel, and are in the re-client repo:
To build a complete set of binaries for reclient with a clangscandeps deps scanner:
$ bazelisk build --config=clangscandeps //:artifacts_tar
[...]
Target //:artifacts_tar up-to-date:
bazel-bin/artifacts.tar
To build a complete set of binaries for reclient with a goma deps scanner:
$ bazelisk build --config=goma //:artifacts_tar
[...]
Target //:artifacts_tar up-to-date:
bazel-bin/artifacts.tar
To install all binaries to a $BINDIR
$ bazelisk run --config=goma //:artifacts_install -- --destdir $BINDIR
[...]
INFO: Running command line: bazel-bin/artifacts_install --destdir $BINDIR
$ bazelisk test //pkg/... //internal/...
[...]
INFO: Elapsed time: 77.166s, Critical Path: 30.24s
INFO: 472 processes: 472 linux-sandbox.
INFO: Build completed successfully, 504 total actions
//internal/pkg/cli:go_default_test PASSED in 0.2s
//internal/pkg/deps:go_default_test PASSED in 1.2s
//internal/pkg/inputprocessor/action/cppcompile:go_default_test PASSED in 0.1s
//internal/pkg/inputprocessor/flagsparser:go_default_test PASSED in 0.1s
//internal/pkg/inputprocessor/pathtranslator:go_default_test PASSED in 0.1s
//internal/pkg/inputprocessor/toolchain:go_default_test PASSED in 0.2s
//internal/pkg/labels:go_default_test PASSED in 0.1s
//internal/pkg/logger:go_default_test PASSED in 0.2s
//internal/pkg/rbeflag:go_default_test PASSED in 0.1s
//internal/pkg/reproxy:go_default_test PASSED in 15.5s
//internal/pkg/rewrapper:go_default_test PASSED in 0.2s
//internal/pkg/stats:go_default_test PASSED in 0.1s
//pkg/cache:go_default_test PASSED in 0.2s
//pkg/cache/singleflightcache:go_default_test PASSED in 0.1s
//pkg/filemetadata:go_default_test PASSED in 2.1s
//pkg/inputprocessor:go_default_test PASSED in 0.2s
Executed 16 out of 16 tests: 16 tests pass.
Reclient can be built to use Goma's input processor. Goma's input processor is 3x faster than clang-scan-deps for a typical compile action in Chrome. Build as follows:
bazelisk build //:artifacts_tar --config=goma
There are four binaries that are built from this repository and used with Android Platform for build acceleration:
These binaries must be stamped with an appropriate version number before they are dropped into Android source for consumption.
We will maintain a consistent version across all of the binaries. That means, when there are changes to only one of the binaries, we will increment the version number for all of them.
In order to be consistent with Semantic versioning scheme, the version format is of the form “X.Y.Z.SHA” denoting “MAJOR.MINOR.PATCH.GIT_SHA”.
Updating version numbers:
MAJOR
MINOR - Update minor version when
PATCH - Update patch version
Release Frequency:
You can update the MAJOR/MINOR/PATCH version numbers by simply changing the
version.bzl
file present in the root of this repository.
Reclient binaries are released into the CIPD (Chrome Infrastructure Package Deployment) with separate packages for Linux, Mac (amd64 and arm64), and Windows. Whenever a new version of Reclient is released there are 2 sets of binaries released for each of the platforms. Those binaries use 2 different include scanners for C++ build actions: clang-scan-deps and goma. The binaries using the goma include scanner have a version number ending with “-gomaip” suffix, the ones using clang-scan-deps don’t have the suffix. Clients migrating from Goma should use the releases using goma include scanner (with -gomaip suffix).
Reclient binaries can be downloaded using CIPD's Web UI, with a CLI client, or using gclient's configuration.
To download Reclient with GomaIP dependency scanner (used for building Chromium):
echo 'infra/rbe/client/${platform}' $RECLIENT_VERSION > /tmp/reclient.ensure
cipd ensure --root $CHECKOUT_DIR --ensure-file /tmp/reclient.ensure
To use Reclient with Clangscandeps (used for Android builds) instead, add -csd
suffix to CIPD package:
echo 'infra/rbe/client/${platform}-csd' $RECLIENT_VERSION > /tmp/reclient.ensure
$RECLIENT_VERSION
- the version of Reclient. It can be set to one of
the following:
re_client_version:0.114.2.81e819b-gomaip
(for Reclient with GomaIP
dependency scanner) or re_client_version:0.114.2.81e819b
(for Reclient
with Clangscandeps).latest
- the latest released Reclient version.stable
- the latest stable Reclient version. Stable version is usually
1-2wks behind the latest
as Reclient needs to run a few days on test
and staging environments without issues and degradations before it's
considered as stable
.$CHECKOUT_DIR
- the location where Reclient should be downloaded.
You can configure
gclient
to download Reclient binaries during the gclient sync
phase. Gclient expects a
DEPS file in the repository’s root directory. The file contains components that
will be checked out during the sync phase. To check out Reclient, the file
should have a similar entry to:
vars = {
...
'reclient_version': '<version>',
...
}
deps = {
...
'<checkout-directory>': {
'packages': [
{
'package': 'infra/rbe/client/${{platform}}',
'version': Var('reclient_version'),
}
],
'dep_type': 'cipd',
},
}
This will instruct gclient to check out <version>
of Reclient from
/infra/rbe/client/<platform>
CIPD package into <checkout-directory>
(example).
Extracting a version to a variable (as in an example above) is optional, but
provides a benefit of being able to override the default value through gclient’s
custom variables.
Note: The snippet above will instruct gclient
to download Reclient with
GomaIP dependency processor. If you prefer Reclient with Clangscandeps, you'd
need to set package to infra/rbe/client/${{platform}}-csd
.
Reclient requires reproxy to be started at the beginning of the build, and
stopped at the end. This is done through bootstrap
binary by executing
following commands:
Start:
bootstrap -re_proxy=$reproxy_location [-cfg=$reproxy_config_location]
Stop:
bootstrap -re_proxy=$reproxy_location -shutdown
Each of Reclient’s binaries can be configured either by command line flags, environment variables, config files, or by combination of either of those (some flags provided in the command line while others in the config file or set as environment variables). If the same flag is defined in the command line and in the config file or as an environment variable, the order of precedence is following (from lowest to highest priority):
To use a configuration file, specify it with the -cfg=$config_file_location
flag. The config file is a list of flag_name=flag_value
pairs, each on a new
line. Example below:
service=$RE_SERVER_ADDRESS
instance=$RE_SERVER_INSTANCE
server_address=unix:///tmp/reproxy.sock
log_dir=/tmp
output_dir=/tmp
proxy_log_dir=/tmp
depsscanner_address=$scandeps_server_location #distributed with Reclient
use_gce_credentials=true
To configure Reclient with environment variables, the variables should be
prefixed with RBE_
(e.g. the value of RBE_service
environment variable is
used to set the service
flag).
Full list of rewrapper config flags can be found in docs/cmd-line-flags.md. A few of the most commonly used flags are:
server_address
flag so that rewrapper and
reproxy can communicate with each other. This value should be UDS on
Linux/Mac (e.g. unix:///tmp/reproxy.sock) and a named pipe on Windows (e.g.
pipe://reproxy.ipc). Depot_tools has a helper choosing the address based on
the platform
(here).type=compile,compiler=clang,lang=cpp
- clang compile actionstype=compile,compiler=clang-cl,lang=cpp
- clang compile actionstype=compile,compiler=nacl,lang=cpp
- nacl compile actionstype=compile,compiler=javac,lang=java
- java compile actionstype=link,tool=clang
- link actionstype=tool
- generic action that doesn’t require any action specific
input processinglocal
, remote
, remote_local_fallback
,
racing
. It is recommended to set it to remote_local_fallback
or
racing
. With remote_local_fallback
it will try to execute the action
remotely and fallback to local if the remote execution failed. With racing
it tries to execute both and picks the one that finished sooner.env_var_allowlist
- List of environment variables allowed to pass to the
proxy. If the build action depends on local environment variables, they
should be set here, so they're reproduced on the remote worker.If you are experiencing sporadic timeouts when dialing reproxy, you might consider adding:
Full list of reproxy flags can be found docs/cmd-line-flags.md. A few of the most commonly used flags are:
localhost:8790
server_address
parameter)scandeps_server
distributed with Reclient set the value to
exec://$absolute_path_to_reclient_dir/scandeps_server
For instance, if
Reclient is checked out to /home/$user/chromium/src/buildtools/reclient/
,
the value of the attribute should be
exec:///home/$user/chromium/src/buildtools/reclient/scandeps_server
If your RE Server implementation does not use RPC authentication then use one of:
true
, do not authenticate with the service
(implied by -service_no_security
).false
, no per-RPC credentials will be used.
Disables --credential_file
, -use_application_default_credentials
, and
-use_gce_credentials
. (default true
).If your RE Server uses RPC authentication then use one of the following flags:
true
(and
--use_application_default_credentials
is false
), use the default GCE
credentials to authenticate with remote execution
(https://cloud.google.com/docs/authentication/provide-credentials-adc#attached-sa).true
, use application default
credentials to connect to remote execution. See
https://cloud.google.com/sdk/gcloud/reference/auth/application-default/login-use_application_default_credentials
and -use_gce_credentials
are false.execrel://
, looks for the credshelper
binary in the same folder as bootstrap/reproxyexperimental_credentials_helper_args - Arguments for the experimental credentials helper, separated by space
The reproxy is typically started via the bootstrap, so it is recommended to
avoid configuring it through the command line flags. It's advised to use
either a configuration file that’s passed to the bootstrap with the -cfg
flag or by setting environment variables before starting the bootstrap
(example).
If you want to collect backend workers' auxiliary metadata (cpu, memory usage per action), you can
generate a .pb (or .proto.bin) file contains the descriptor information that will be used by reproxy at runtime to
decode the auxiliary metadata, which is a proto message in the type of google.protobuf.Any
.
Once you have customized auxiliary_metadata.proto
file per your backend worker's
specification, compile it as a .pb
or .proto.bin
file with protoc
, and pass
the file path to reproxy via --auxiliary_metadata_path
flag, or environment
variable RBE_auxiliary_metadata_path
. Then, at runtime, reporxy will use this file to parse
your backend worker's auxiliary metadata and log the data into reporxy logs.
cd api/auxiliary_metadata # or where you have the cusotmized `.proto` file
protoc \
--proto_path=. \
--descriptor_set_out=auxiliary_metadata.pb \
auxiliary_metadata.proto
export RBE_auxiliary_metadata_path=~/Workspace/re-client/api/auxiliary_metadata/auxiliary_metadata.pb
# then continue with your regular build with reproxy
or
cd api/auxiliary_metadata # or where you have the cusotmized `.proto` file
protoc \
--proto_path=. \
--descriptor_set_out=auxiliary_metadata.proto.bin \
auxiliary_metadata.proto
export RBE_auxiliary_metadata_path=~/Workspace/re-client/api/auxiliary_metadata/auxiliary_metadata.proto.bin
# then continue with your regular build with reproxy
It's worth noting that the backend can give this proto message any arbitrary name;
however, the client side proto message should strictly use AuxiliaryMetadata
to receive it.
For example, in the unit test, the backend send the proto msg out with name WorkerAuxiliaryMetadata
, and client receives it as AuxiliaryMetadata
.
To execute your build actions remotely through Reclient, the build command should be prepended with:
$rewrapper [-cfg=$config-file] -exec_root=$checkout-dir --
, where:
When rewrapper is executed, it passes the build command to a running instance of reproxy that:
Before the build is executed, reproxy needs to be started by bootstrap and shut down at the end of the build.
During the run, reproxy writes its application-level logs to a directory
specified by a log_dir
flag and logs records about the executed build actions
to an RPL file in a directory specified by the proxy_log_dir
flag. During
reproxy shutdown, bootstrap dumps Reclient related build metrics to
rbe_metrics.txt
file saved at a location specified by the bootstrap's
output_path
flag.
GN is a meta-build system that generates build files for Ninja. Its configuration files are written in a simple, dynamically typed language. Reclient can be integrated with the build by modifying the GN config files. Because of GN's language flexibility the method of how Reclient should be integrated will depend on the project, but usually it should involve adding a rewrapper prefix (example) that's controlled by a gn argument (example), and starting and stopping reproxy before and after the build. The latter might be done by a helper script with reproxy start and stop steps around the ninja call example.
You can integrate CMake with Reclient by using
<LANG>_COMPILER_LAUNCHER
property. This property is initialized by the value of the
CMAKE_<LANG>_COMPILER_LAUNCHER
variable if it is set when a target is created.
For instance, to use Reclient for c/c++ compile actions, you’d need to set both
CMAKE_C_COMPILER_LAUNCHER
and CMAKE_CXX_COMPILER_LAUNCHER
to
$rewrapper;-cfg=$config-file;-exec_root=$checkout-dir
(the property accepts
semicolon separated list as a launcher command).
Please note that CMake operates on absolute paths and you need to ensure that RE
server executes the action on a remote worker in the same directory as it is in
a local build machine (the method depends on your RE Server implementation).
Moreover, please be aware that rewrapper's canonicalize_working_dir
flag
tampers the build actions' inputs paths, and thus should be disabled for the
build actions generated by CMake.
Example of CMake build integration with Reclient can be found here.