Introduce tracking of individual sources, using the Remote Asset API

[BuildStream-Migration-Bot]

See original issue on GitLab In GitLab by [Gitlab user @sstriker] on Mar 25, 2020, 22:08

We can reduce the load and reliance on additional services by leveraging the Remote Asset API. For example, instead of having all clients poll git services, the FetchService.FetchDirectory API is used to resolve the commit at a certain branch. As clients all track to the same revision, in cache hits are more likely for sources, artifacts and actions.

To support this we need to extend the Source Plugin API to return the list of URIs and qualifiers as needed by the FetchService. Specifically:

• FetchDirectoryRequest.uris is the complete set of URLs that represent the content of the source. This is the full set after alias expansion. For example: git.example.com/foo/bar.git and git-mirror.example.com/foo/bar.git.
• FetchDirectoryRequest.qualifiers is the minimal set of qualifiers that identifies the source. This must exclude the commit for the branch. For example:
  • vcs.branch = master
  • resource_type = application/x-git

In the response the client will learn the Digest of the source as well as all other qualifiers the service knows about. This would include identifying information the source plugin would use in its ref. For example:

• FetchDirectoryResponse.uri is the URL that matched, that represent the content of the source. For example: git.example.com/foo/bar.git.
• FetchDirectoryResponse.qualifiers is the complete set of qualifiers associated with the source. For example:
  • vcs.commit = b5123b1bb2853393c7b9aa43236db924d7e32d61
  • resource_type = application/x-git
  • vcs.branch = master

Behavior should be configurable to support the following use cases:

• client does not use the Remote Asset API to track sources, and only uses the source plugin native track
• client uses the Remote Asset API to track sources, and falls back to using the source plugin native track
• client uses the Remote Asset API to track sources, and does not fall back to using the source plugin native track
• client uses the Remote Asset API to push sources, after using the source plugin native track and fetch
• client does not use the Remote Asset API to push sources
• client is configured to only accept results younger than a certain age.

See also:https://mail.gnome.org/archives/buildstream-list/2020-February/msg00000.html

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on Mar 25, 2020, 22:08

Given the Source Plugin API changes suggesting this for 2.0

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @cs-shadow] on Mar 26, 2020, 12:18

[Gitlab user @sstriker] thanks for the write-up.

How do you envision this working for SourceTransforms (like pip source etc) that require access to other sources listed for that element? Or would such sources have to resort back to how they are handled now?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on Mar 26, 2020, 21:28

Excellent question.

It really depends on how specialized of a source plugin we are willing to make. And how much there is to gain when tracking these types of sources - in the pip source case, I imagine there might be quite a bit of work that can be avoided. That is, going from a requirements.txt to a frozen set of requirements.

I imagine that during tracking source plugins that have indicated they need the previous sources, to be passed the Tree (vDirectory?) of the previous sources at the start of tracking. The pip source plugin could use the digest of the requirements.txt file as a qualifier, and then maybe use these for tracking:

• buildstream.build.plugins.pip.requirements_digest = digest
• buildstream.build.plugins.pip.additional_packages = package1,package2,...

The ref for a pip source plugin is the frozen requirements, I think that would result in push qualifiers would be:

• buildstream.build.plugins.pip.requirements_digest = digest
• buildstream.build.plugins.pip.additional_packages = package1,package2,...
• buildstream.build.plugins.pip.frozen = frozen_requirements

In a setup like this, you would have a central BuildStream instance be taking care of the tracking:

• whenever the source changes, and/or
• periodically, as the requirements.txt may resolve to a new frozen set, and/or
• triggered because it is known that a package in requirements.txt for this project has a new version

And each other client would be tracking to the same set of frozen deps the central instance has. For the pip source plugin this has two implications:

• BuildStream clients will only spend time doing native tracking when the requirements.txt or element configuration changes for this source
  • BuildStream clients will only then need file content of the previous sources
  • BuildStream clients will only then hit pypi endpoints

In short, I think that there might be source plugins that require previous sources, where it makes sense to resort to native tracking. In other cases, the specialized handling may be worth it.

Make sense?

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @sstriker] on Mar 27, 2020, 20:51

[Gitlab user @cs-shadow]: After a night's sleep, an updated answer

A not unimportant benefit I left out for this pip source example:

• BuildStream clients wouldn't need to have python host tools to track (caveat: only when not falling back to native tracking)

Now, if we wanted to make the example of pip source more generic, we could document qualifiers in the Remote Asset API spec.

Let's assume that for the Remote Asset API in combination with PyPI, we use the following convention:

• We use FetchDirectory/PushDirectory
• The root_directory_digest refers to a directory in CAS with all the downloaded sdists from PyPI
• The uris are the PyPI index URLs we can retrieve these sdists from
• We have the following generalized qualifiers
  • pypa.pip.requirements: sorted list of packages with optional version information, eg. package1,package2,package3==0.5.4
  • pypa.pip.contraints: sorted list of packages with version information, eg. package1==0.1.2,package2==2.4.0
  • pypa.pip.requirements.frozen: sorted list of packages with version information, eg. package1==0.1.2,package2==2.4.0,package3==0.5.4
• We have the following BuildStream specific qualifiers (these may be made generic as well, open to suggestions)
  • buildstream.build.plugins.pip.requirements.digest: digest of the requirements.txt
  • buildstream.build.plugins.pip.constraints.digest: digest of constraints.txt

fetch

On a bst fetch operation we already have a "ref", which we map to the pypa.pip.requirements.frozen qualifier. We call FetchDirectory with only that qualifier.

track

On a bst track operation we want the following to happen.

1. We attempt FetchDirectory with qualifiers specialized for previous sources:
   • buildstream.build.plugins.pip.requirements.digest
   • buildstream.build.plugins.pip.constraints.digest If we get a result back we're done, and can use the returned pypa.pip.requirements.frozen as ref
2. We attempt FetchDirectory with general qualifiers:
   • pypa.pip.requirements
   • pypa.pip.contraints For this to work we do actually need to get the file contents for requirements.txt and constraints.txt. These should be available from CAS, and auto-hydrated when accessed through vDirectory(?). If we get a result back we're done, and can use the returned pypa.pip.requirements.frozen as ref
3. We fall back to native tracking

As you can see this reveals we do need a protocol for the Source Plugin API in combination with Remote Asset tracking and previous sources. Step 1 would not be relevant to plugins that don't have previous sources.

push

On a bst push operation we send all of the qualifiers:

• pypa.pip.requirements
• pypa.pip.contraints
• buildstream.build.plugins.pip.requirements.digest
• buildstream.build.plugins.pip.constraints.digest
• pypa.pip.requirements.frozen

[BuildStream-Migration-Bot]

In GitLab by [Gitlab user @cs-shadow] on Apr 1, 2020, 22:39

Many thanks for the detailed response.

The overall plan seems good to me. On the high-level design, I only have one comment/quesiton.

I'm unsure about adding the input requirements as a qualifier. When tracking the same exact same requirements at different times, we are not guaranteed the same result. This will depend on how often the dependencies change, but will see heavy churn with lots of unbounded dependencies.

This is because pip will pick the latest version each time. So, if there is a new release of any dependency between two track operations, the output will be different.

Imagine this scenario:

1. Package Ponies releases 1.0.
2. User had Ponies >= 1.0 as their only requirement in element Unicorn.
3. User tracks Unicorn and pushes Ponies to the cache.
4. Ponies now releases 2.0.

Now, if we use pypa.pip.requirements as a qualifier, we will get back the cached version (version 1.0) in form of pypa.pip.requirements.frozen. However, if the user does native tracking at this point, they will get version 2.0. As such, it is not good for reproducibility.

This is not a general problem with the plan itself, but specifically with pip source. However I think it may extend to other popular package managers as well, since most of them pick the latest available version.

Having said that some other package managers (like the one in Go) aim to provide this guarantee by picking the oldest allowed version.

---

A couple of minor comments:

BuildStream clients wouldn't need to have python host tools to track (caveat: only when not falling back to native tracking)

This is pretty neat.

sorted list of packages with optional version information, eg. package1,package2,package3==0.5.4

I don't think it matters here but I'd just mention it that we will likely have duplicates in this list. When merging differnet requirement files and inline requirements, BuildStream relies on pip's logic to satisfy all the constraints. So, a single packge may appear multiple times, like (package1,package1>0.1 etc).

pypa.pip.contraints: sorted list of packages with version information

Maybe I'm missing something but I'm not sure if I understand what are you referring to by "constraints" here.

The way I understand it, pypa.pip.requirements is the set of input requirements and pypa.pip.requirements.frozen is the result of tracking on the input requirements. What are these constraints then?

Are they just additional input, or something else?