Not possible to use generic servers implementing the remote-apis as remote cache #1050

Closed BuildStream-Migration-Bot closed 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @jjardon] on Jun 23, 2019, 18:28

Summary

Builstream seems to complain with the remote-apis specs (https://github.com/bazelbuild/remote-apis), but still seems to be not possible to use some of the servers that implement those apis as remote cache

What is the current bug behavior?

It's not possible to use remote-apis servers like buildfarn and buildbarn as remote cache

What is the expected correct behavior?

It should be possible to use servers that implement the remote-apis as remote cache

In GitLab by [Gitlab user @jjardon] on Jun 23, 2019, 18:29

marked this issue as related to #1037

In GitLab by [Gitlab user @tpollard] on Jun 25, 2019, 14:57

For the caching of buildstream artifacts, I think this behaviour it expected since the transition to proto based artifacts?

In GitLab by [Gitlab user @jjardon] on Jun 25, 2019, 16:35

[Gitlab user @tpollard] can you confirm it's a intended decision not to be compatible (for caching only) with other remote-apis' server implementations?

In GitLab by [Gitlab user @raoul].hidalgocharman on Jun 25, 2019, 16:51

This is currently the expected behaviour, but once #1041 is implemented we should be able to to specify generic remote-api's along with buildstream specific services.

In GitLab by [Gitlab user @juergbi] on Jun 25, 2019, 16:54

This is currently the expected behaviour, but once #1041 is implemented we should be able to to specify generic remote-api's along with buildstream specific services.

Yes, I was just about to comment the same. To clarify further, you'll still need bst-artifact-server or another implementation of the artifact service for artifact caching. However, the actual blobs can then be cached on any CAS server.

In GitLab by [Gitlab user @jjardon] on Jun 26, 2019, 03:10

[Gitlab user @raoul].hidalgocharman [Gitlab user @juergbi] thanks for the clarification

so how this will work if I only want an artifact cache using buildbarn, for example? From you response seems I have to setup:

buildbarn for CAS and AC (Action Cache)
bst-artifact-server for artifact caching (Why is this needed at all (I'm ok if you redirect me to some docs)?)

Is this correct or I'm missing something?

In GitLab by [Gitlab user @juergbi] on Jun 26, 2019, 08:29

buildbarn for CAS and AC (Action Cache)

CAS, yes. Action cache is irrelevant if you don't use remote execution.

bst-artifact-server for artifact caching (Why is this needed at all (I'm ok if you redirect me to some docs)?)

Yes, but please note that bst-artifact-server without CAS is a very simple and lightweight service and shouldn't need lots of resources.

CAS on its own is not sufficient for artifact caching as clients can only retrieve files and directories with a known digest. I.e., you can't retrieve the build result unless you already know the digest of the build result (CAS doesn't support anything like git/ostree refs).

The action cache provides mappings from action descriptions to action results. However, it's not compatible with BuildStream's cache key approach. The two large conceptual differences are:

BuildStream cache keys are mainly derived from the cache keys of the build dependencies, the sources and the commands. The REAPI Action message is mainly based on the input tree and a command. For BuildStream remote execution the input tree includes the staged build dependencies and the sources. This means that BuildStream would have to (virtually) stage the build dependencies and sources to generate the Action message to be able to query the action cache, and that's a lot more expensive than querying the bst artifact service using the BuildStream cache key. There are additional issues such as the build of a single bst element may require multiple REAPI Actions (e.g., integration commands in separate action).
BuildStream has optional support for reusing a cached artifact even when its build dependencies have been updated (non-strict build plans). The action cache doesn't allow this.

That said, it might be possible to change BuildStream to always generate Action messages (i.e., also for local builds) and then have a local cache for mapping BuildStream cache keys to Action messages (to avoid having to stage the build dependencies all the time). This could allow using a remote action cache without remote execution and without a remote artifact service. However, this would make local builds more expensive (when a build requires multiple REAPI Actions, you have to store the result of each of those), you'd lose some of the benefits of non-strict build plans, and remote cache lookups would be slower whenever the cache key changes (as BuildStream would then have to stage the build dependencies and sources for the remote cache lookup).

Theoretically, we could also abuse the action cache with a magic bst command and a BuildStream cache key in the Action message, but I don't think we want to go down that road.

In GitLab by [Gitlab user @sstriker] on Jun 26, 2019, 14:51

[Gitlab user @juergbi] this is a really clear explanation, we should include it in the docs somewhere.

That said, it might be possible to change BuildStream to always generate Action messages (i.e., also for local builds) ... and remote cache lookups would be slower whenever the cache key changes (as BuildStream would then have to stage the build dependencies and sources for the remote cache lookup).

Interesting idea. Performance may actually be ok, that would be something to test. This is not part of any near term plans though, and unless someone is really passionate about it I don't see it happening this year.

Theoretically, we could also abuse the action cache with a magic bst command and a BuildStream cache key in the Action message, but I don't think we want to go down that road.

Ew, no.

In GitLab by [Gitlab user @jjardon] on Jun 27, 2019, 00:59

Thanks [Gitlab user @juergbi] for your detailed reply. I have some questions though

CAS, yes. Action cache is irrelevant if you don't use remote execution.

mmm, maybe there is a nomenclature problem here, but we have been using buildbarn as remote cache using both CAS and AC (using bazel as a client). Basically using https://github.com/buildbarn/bb-storage only

The action cache provides mappings from action descriptions to action results. However, it's not compatible with BuildStream's cache key approach

Oh, so are you saying that AC is not needed at all, or It's irrelevant on the buildstream specific use case? Maybe I'm misunderstanding something or we are using nomenclature in a different way but I'd like to clarify this

In GitLab by [Gitlab user @juergbi] on Jun 27, 2019, 05:08

Oh, so are you saying that AC is not needed at all, or It's irrelevant on the buildstream specific use case? Maybe I'm misunderstanding something or we are using nomenclature in a different way but I'd like to clarify this

Bazel uses AC for remote caching. However, BuildStream can't use AC for local builds without the relatively large changes I described after the two conceptual differences.

I.e., AC is not irrelevant for Bazel clients but it's irrelevant for BuildStream for remote caching of local builds, even after #1041 is implemented.

In GitLab by [Gitlab user @tristanvb] on Jun 27, 2019, 06:36

Oh, so are you saying that AC is not needed at all, or It's irrelevant on the buildstream specific use case? Maybe I'm misunderstanding something or we are using nomenclature in a different way but I'd like to clarify this

Bazel uses AC for remote caching. However, BuildStream can't use AC for local builds without the relatively large changes I described after the two conceptual differences.

I.e., AC is not irrelevant for Bazel clients but it's irrelevant for BuildStream for remote caching of local builds, even after #1041 is implemented.

Whoa, this whole thread sounds ultra confusing to me.

I think it's simpler to just say that:

BuildStream needs an ArtifactCache (which is more than just a CAS) to cache artifacts, regardless of whether you are building locally or remotely
- Or as stated in previous messages, a ~~hypothetical~~ implementation of something where a third party CAS is used in conjunction with a small service that just takes care of the artifact protos and ref translations ~~(this does not exist, and I see no statement of intent towards creating such a thing, so lets be clear that this is not a thing that users can use).~~
EDIT: Sorry, [Gitlab user @juergbi] pointed out to me that there is in fact a plan to do this which #1041 is a part of
BuildStream uses an ActionCache for remote execution, this is rather transient state any only related to creating artifacts but not storing them
- There is a hair brained idea to use the ActionCache for artifact storing... or something along those lines, but that wont happen, and just talking about it just makes this whole topic less informative and rather confusing instead :)

This is all made even more confusing by the fact that "AC" obviously stands for ArtifactCache, but now also stands for ActionCache.

In GitLab by [Gitlab user @sstriker] on Jun 27, 2019, 10:29

[Gitlab user @tristanvb], I'm pretty sure hair-brained ideas are awesome :) We should revisit this one at some later point in time.

This is all made even more confusing by the fact that "AC" obviously stands for ArtifactCache, but now also stands for ActionCache.

AC has always stood for ActionCache, we have been pretty deliberate in not abbreviating ArtifactCache that way.

In GitLab by [Gitlab user @jjardon] on Jun 27, 2019, 13:48

mentioned in commit remote-apis-testing/remote-apis-testing[Gitlab user @2c28598a119148773412f4e924793031322e9222]

In GitLab by [Gitlab user @jjardon] on Jun 27, 2019, 13:49

mentioned in merge request remote-apis-testing/remote-apis-testing!49

In GitLab by [Gitlab user @jjardon] on Jun 27, 2019, 13:51

Thanks everyone for your inputs. I think we can summarize as following (please correct me if wrong)

As today, builstream is not compatible with other generic remote-apis servers (only with BuildGrid)
In the future, it will be possible to combine a generic CAS, with buildstream's specific ArtifactCache

Is this accurate?

In GitLab by [Gitlab user @jjardon] on Jul 1, 2019, 17:06

marked this issue as related to #1041

In GitLab by [Gitlab user @Qinusty] on Jul 26, 2019, 16:59

mentioned in commit remote-apis-testing/remote-apis-testing[Gitlab user @15f1e6e6f598b2447b78cd3875adf18e1b063e16]

In GitLab by [Gitlab user @jjardon] on Aug 30, 2019, 12:07

marked this issue as related to #1114

In GitLab by [Gitlab user @jjardon] on Sep 16, 2019, 15:33

This should work now after !1540 has been merged

In GitLab by [Gitlab user @jjardon] on Sep 16, 2019, 15:33

closed

apache / buildstream