Improve compatibility of Buildstream with Buildbarn and Buildfarm #1114

Open BuildStream-Migration-Bot opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @Qinusty] on Aug 30, 2019, 11:13

Background

Buildstream is currently not compatible with Buildbarn, Buildfarm or RBE.

Task description

Non exhaustive list of tasks

[x] Merge https://gitlab.com/BuildStream/buildstream/merge_requests/1540 to support separate endpoints for remote caching and resolve https://gitlab.com/BuildStream/buildstream/issues/1050
Address Buildbarn issues:
- [ ] buildbarn-RE#22 Summary issue for Buildstream compatibility issues.
Address Buildfarm issues:
- [ ] Investigate https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/43

Discussion

With the goal of improving Buildstream compatibility with alternative RE server implementations aside from the tested buildgrid. Buildstream requires the extension of bst-artifact-server to work with remote execution (RE) services which follow the RE spec. This is fine as the overhead is not too large on the artifact server and https://gitlab.com/BuildStream/buildstream/merge_requests/1540 will allow for separate endpoints to be used for the CAS and index server instances.

Buildbarn

Testing compatibility with Buildbarn is detailed and it is clear that Buildbarn is not following the spec. Buildbarn will have to correct the implementation of output_directories to be relative to the working_directory which is currently (correctly) relative to the input root. This is an unseen issue when Buildbarn interacts with Bazel as builds appear to specify the working_directory as . which is relative to the input_root. Following these changes, Buildstream should be compatible with Buildbarn given an appropriate image for sandboxing as detailed in slack.

Mickaël Carl 3:47 PM
--------------------
So indeed, bb-runner is currently provided on top of a Ubuntu image that’s also in use by RBE if I’m not mistaken
You are of course free to decide for yourself which base to use and move to a different base image

It’s actually incredibly simple, see https://github.com/buildbarn/bb-remote-execution/blob/master/cmd/bb_runner/BUILD.bazel

Specifically line 41

Buildfarm

Buildfarm also causes issues while working with Buildstream however I am currently unsure of whether this an issue with Buildstream or Buildfarm. https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/43 details how to reproduce the issues I found and provides a stack trace pointing to an issue while uploading results. It is likely that Buildstream interacts with Buildfarm in a way that has gone unnoticed whilst Bazel has been used with Buildfarm.

In GitLab by [Gitlab user @Qinusty] on Aug 30, 2019, 11:55

changed the description

In GitLab by [Gitlab user @jjardon] on Aug 30, 2019, 12:07

marked this issue as related to #1050

In GitLab by [Gitlab user @Qinusty] on Aug 30, 2019, 16:29

changed the description

In GitLab by [Gitlab user @Qinusty] on Sep 5, 2019, 10:30

changed the description

In GitLab by [Gitlab user @BenjaminSchubert] on Sep 24, 2019, 10:56

marked the task Merge https://gitlab.com/BuildStream/buildstream/merge_requests/1540 to support separate endpoints for remote caching and resolve https://gitlab.com/BuildStream/buildstream/issues/1050 as completed

In GitLab by [Gitlab user @christopherphang] on Jan 30, 2020, 23:45

Buildbarn/buildstream compatability is currently a major goal of the Celduin project.

See https://gitlab.com/celduin/remote-execution/remote-execution/issues/1. This will setup buildbarn, in conjunction with the artifact server, to allow buildbarn to act as a cache service.

Once we have stable infrastructure, we'll use this as a basis to deploy bb-workers that can then be used for remote-execution

In GitLab by [Gitlab user @coldtom] on Jan 31, 2020, 16:22

We have a Kubernetes deployment of a remote artifact cache with bb-storage CAS at https://gitlab.com/celduin/infrastructure/celduin-infra/merge_requests/5.

This is a cache suitable to be public facing, with pull available for everyone and push requiring SSL authentication. One slight issue with it is that when attempting to push illegally BuildStream fails saying "Unhandled RPC errors: 7". Do people think that Permission Denied gRPC responses should be handled by BuildStream more gracefully?

In GitLab by [Gitlab user @jjardon] on Feb 2, 2020, 08:49

marked this issue as related to remote-apis-testing/remote-apis-testing#20

In GitLab by [Gitlab user @jjardon] on Feb 2, 2020, 08:49

marked this issue as related to remote-apis-testing/remote-apis-testing#56

In GitLab by [Gitlab user @BenjaminSchubert] on Feb 2, 2020, 10:01

Definitely, having to deal with gRPC errors is not a good experience and we should at least be handling every common use cases.

In GitLab by [Gitlab user @coldtom] on Feb 24, 2020, 12:17

One of the blockers to get BuildStream working with BuildBarn for Remote Execution has been resolved: https://github.com/buildbarn/bb-remote-execution/issues/18

I don't know if anyone has tried running the two together yet, though.

In GitLab by [Gitlab user @coldtom] on Mar 5, 2020, 09:52

[Gitlab user @scott].clarke made a great write up of issues encountered when trying to use BuildBarn as the remote executor for BuildStream here https://gitlab.com/remote-apis-testing/remote-apis-testing/issues/12#note_298649889.

In GitLab by [Gitlab user @tpollard] on Nov 18, 2020, 12:33

Buildstream can be seen working with Buildbarn at https://gitlab.com/remote-apis-testing/remote-apis-testing/-/merge_requests/192

This depends on https://github.com/apache/buildstream/issues/1313

apache / buildstream