Add ability to build without interacting with available artifact servers #752

Open Cynical-Optimist opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @tpollard] on Nov 7, 2018, 14:19

Background

As mentioned in #buildstream, it seems reasonable that bst build should have the option to build everything from source without attempting to reach out to remote caches, especially when consuming junctions in which artifact servers are defined and populated for pulling or have the opportunity to be pushed to. One example of wanting to do this would be to periodically check that a whole project can be built from 'scratch'.

Task description

Add a cli option (could be main or specific to build) and a user config context option with a suitable name for this behaviour override
Use cli/context/app as appropriate to consume this during streams creation of queues, emitting pull and push queues as appropriate
Add tests

Acceptance Criteria

Testable and does not break normal behaviour

In GitLab by [Gitlab user @tpollard] on Nov 7, 2018, 14:21

The use of overriding fetchers and pushers to 0 to handle this situation is also a possibility

In GitLab by [Gitlab user @tpollard] on Nov 7, 2018, 14:22

changed the description

In GitLab by [Gitlab user @toscalix] on Nov 12, 2018, 09:46

assigned to [Gitlab user @tpollard] and [Gitlab user @valentindavid]

In GitLab by [Gitlab user @tristanvb] on Nov 12, 2018, 09:46

A constant goal is to have a minimal API surface at all times, so when deciding on what new things to expose to the CLI and user configuration, we should also consider:

People may want to ignore project configured artifact caches and only observe user configured artifact servers
People may want to specify a single URI to override all artifact server URIs

In GitLab by [Gitlab user @tpollard] on Nov 12, 2018, 11:02

I agree both of those situations are definitely worth considering [Gitlab user @tristanvb], good points

In GitLab by [Gitlab user @tpollard] on Nov 12, 2018, 13:17

I'm going to give this a go whilst I have some overhead time

In GitLab by [Gitlab user @tpollard] on Nov 13, 2018, 17:17

mentioned in merge request !950

In GitLab by [Gitlab user @tpollard] on Nov 13, 2018, 17:19

I've submitted a WIP for this, it's lacking new tests but I hope it captures the original idea and further scope raised

In GitLab by [Gitlab user @tpollard] on Nov 13, 2018, 17:45

I think as a follow up from this it would be beneficial to add a --remote option to build, to match pull & push to specific remotes

In GitLab by [Gitlab user @tpollard] on Nov 14, 2018, 17:52

I've taken !950 out of WIP

In GitLab by [Gitlab user @toscalix] on Dec 17, 2018, 09:29

marked this issue as related to #477

In GitLab by [Gitlab user @tpollard] on Feb 5, 2019, 10:28

At the recent BuildStream gathering we discussed possible use cases involving limiting certain actions to certain remotes. As a base this MR coves splitting all actions into user, project, or no remotes. This should be extended to differentiate project remotes into the top level project and sub projects (i.e junctions). It would also be beneficial to potentially have more granular control over remotes per command type (i.e push/pull, specifying different config for each)

buildstream-migration / bst-staging