Allow staging part of a source #493

Open BuildStream-Migration-Bot opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @tlater] on Jul 23, 2018, 13:54

Background

Sometimes projects are structured in a way that necessitates singling out a subdirectory before building the project. This happens quite frequently in projects that are an amalgamate of various plugins or themes.

Currently, to build one of these from any source besides tar, local or zip, we need to prefix configure-commands with the relevant commands to move this around - be that mv-ing, cp-ing, cd-ing or suchlike.

This can be annoying for elements that would otherwise not have to depend on an element that contains these binaries (this is fairly common for themes).

Task description

To mitigate this, we could make the base-dir API from our zip/git global (and perhaps rename it).

Acceptance Criteria

We should have a way to stage only part of a source in all source plugins.

In GitLab by [Gitlab user @tristanvb] on Jul 23, 2018, 14:01

I think this is not entirely accurate, the tar and zip plugins allow this with their base-dir option for the reasons you describe (one use case that I think is used, is extracting only the spidermonkey JS from a mozilla tarball).

Note also, the directory configuration is a Source level configuration handled in the core, plugins are not allowed to use this for local configuration (however Source plugins do include it in their documentation), it works the same way for every plugin and describes where the data should be staged in the sandbox, not what part of the Source data should be staged at all.

Whether this is relevant for git, bzr, local or other sources is questionable I think; if it's requested for a use case out there, we should probably add functionality for this on plugins where it is required.

In GitLab by [Gitlab user @tlater] on Jul 23, 2018, 14:37

[Gitlab user @tristanvb] Gah, my mind must have slipped, I meant base-dir, not directory. Correcting that.

I was just asked if this was possible on git sources, and I've come across it on a few projects myself - e.g. StumpWM (although they have since split off contrib into a separate project), and in fact our very own bst-here.

I rarely use plugins besides git, so I'm not certain that it's required elsewhere, but project-in-subdirectory seems to be a common pattern that isn't tied to source implementation, so surely we should have generic API in place?

In GitLab by [Gitlab user @tlater] on Jul 23, 2018, 14:37

changed the description

In GitLab by [Gitlab user @tristanvb] on Jul 23, 2018, 14:55

so surely we should have generic API in place?

Yeah that does makes sense, however it will surely amount to different, per-plugin implementations.

Perhaps we could have the base Source class have a hacky fallback for convenience, but the plugin should normally be implementing this and reducing I/O based on this.

apache / buildstream