Ability to manually optimize build graph #942

Open BuildStream-Migration-Bot opened 3 years ago

See original issue on GitLab In GitLab by [Gitlab user @nanonyme] on Mar 3, 2019, 14:21

This is mostly a request for comments at this point. There's some exceptional build items (read: take very long to compile) in freedesktop-sdk like LLVM that are the reverse dependencies of many things and as a consequence you'd want to to start compiling them as early as possible. In other words, not all elements are really equal.

Should BuildStream expose some tools (eg element priority) to projects to be able to have more control over element build order? Is it already in current element model possible to affect that without adding artificial dependencies?

In GitLab by [Gitlab user @nanonyme] on Mar 3, 2019, 14:22

changed the description

In GitLab by [Gitlab user @tristanvb] on Mar 9, 2019, 09:03

Some similar discussion happened at our gathering, or rather discussion around dictating the required resources needed in order to successfully build an element (see issue #185, I think this is related).

We discussed how Bazel currently handles this, which iirc is to say that a build unit requires 2GB of free memory and only uses a single process (this works for Bazel because they generally farm out very granular commands, not builds).

By the same logic, we might calculate a build in terms of abstract units and restrict parallelism in those builds to the amount of units required by an element or similar - I think in any case something declarative is required, and the same declaration which allows us to ensure enough resources are present to run a build should be sufficient to deduce how heavy a build is and possibly we can reorganize the scheduling order to treat heavier builds first whenever possible.

In GitLab by [Gitlab user @tristanvb] on Mar 9, 2019, 09:03

marked this issue as related to #185

In GitLab by [Gitlab user @nanonyme] on Mar 9, 2019, 20:27

Yeah, I think this is related. Something that changed with freedesktop-sdk 18.08 closer to New Year is that we now build not one but two LLVM's (both 6 and 7). It's a very clear and significant resource bottleneck somewhere near the middle of the build trying to build these two. LLVM 7 also is dependency for eg Mesa, gstreamer etc.

apache / buildstream

Ability to manually optimize build graph #942