Introduce caching of individual sources, using a Remote Asset API based Asset Cache #1274

Open Cynical-Optimist opened 4 years ago

See original issue on GitLab In GitLab by [Gitlab user @sstriker] on Mar 25, 2020, 22:02

We can reduce the load and reliance on additional services by leveraging the Remote Asset API. CAS and Remote Asset API combined can serve as a cache of the content, with the additional benefit that having the content in CAS means it can be referred to in Remote Execution without additional uploads.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:

FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that uniquely identifies the source. It's expected to be closely tied to the value of ref. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git

We will need to expand the Source Plugin API similarly to return the list of URIs and qualifiers as needed by the PushService. Specifically:

PushDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
PushDirectoryRequest.qualifiers is the complete set of qualifiers associated with the source. For example:
- vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
- resource_type = application/x-git
- vcs.branch = master

Behavior should be configurable to support the following use cases:

client does not use the Remote Asset API to fetch sources, and only uses the source plugin native fetch
client uses the Remote Asset API to fetch sources, and falls back to using the source plugin native fetch
client uses the Remote Asset API to fetch sources, and does not fall back to using the source plugin native fetch
client uses the Remote Asset API to push sources, after using the source plugin native fetch
client does not use the Remote Asset API to push sources

See also: https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

In GitLab by [Gitlab user @sstriker] on Mar 25, 2020, 22:03

Given the Source Plugin API changes suggesting this for 2.0

In GitLab by [Gitlab user @coldtom] on Aug 10, 2020, 14:56

I notice that in this outline different sets of qualifiers are used for fetch/push, which may cause problems with some servers which obey the Remote Asset API specification, in particular if a server opts in for the "MAY" here. It is probably fine to say that BuildStream requires the ability to match on a subset of qualifiers, but this does mean that there may be Remote Asset API server implementations that are not BuildStream compatible.

In GitLab by [Gitlab user @juergbi] on Aug 25, 2020, 15:45

mentioned in commit ab8316b37591e044a69276ec644b6f1a74c986eb

In GitLab by [Gitlab user @juergbi] on Aug 25, 2020, 15:45

mentioned in commit 8047bd78dfd2607a53b615b33dd7881068a3f2cb

In GitLab by [Gitlab user @juergbi] on Aug 25, 2020, 15:53

mentioned in commit 01f7621cea149dd97d4e84c502cc30c7ee166d39

In GitLab by [Gitlab user @juergbi] on Aug 25, 2020, 15:58

mentioned in merge request !2044

In GitLab by [Gitlab user @juergbi] on Aug 25, 2020, 17:08

mentioned in commit abcbeab5741044da897b1d14154faf4db3432582

In GitLab by [Gitlab user @juergbi] on Sep 3, 2020, 10:10

mentioned in commit 35b784099a1f14fa80123d6d2436c9be2db749f5

In GitLab by [Gitlab user @juergbi] on Sep 3, 2020, 10:10

mentioned in commit 94482a04c1b2726fec5ad3d49a8b532e3f41197e

In GitLab by [Gitlab user @juergbi] on Sep 3, 2020, 13:12

mentioned in commit 7df5e40e8673b8a59a93cfdf2685cf120d6a98db

In GitLab by [Gitlab user @juergbi] on Sep 3, 2020, 13:12

mentioned in commit 7175dbb76aab99935a4e3f5884bac9451bfb655e

buildstream-migration / bst-staging

Introduce caching of individual sources, using a Remote Asset API based Asset Cache #1274