oe-lite / core

Official OE-lite/core repository - moved to GitLab.com
https://gitlab.com/oe-lite/core
Other
4 stars 17 forks source link

lib/oelite: alternate universe implementation #207

Closed Villemoes closed 7 years ago

Villemoes commented 7 years ago

The way "oe bake world" and "oe bake universe" is currently implemented means that the buildbot spends a lot of time building stuff that is already known to work.

The problem is (roughly) that when a recipe is a "primary" target, whether to build it again relies on whether the current do_build stamp file matches the hash we compute for the do_build task, and the files in the stamp directory reflect (more or less) the status of $WORKDIR. So when a PR on meta/core updates gcc [1] the buildbot does a complete rebuild of everything, which is of course what we want.

However, when another PR comes in which just corrects a wrong LICENSE in some meta/base recipe, we (of course) use the master branch from meta/core. Since all the do_build stamp files don't match, we build everything once again, despite the fact that $PACKAGE_DEPLOY_DIR in all likelyhood contains tar-balls with the appropriate hash values from the second-to-last build.

So, instead of making every (compatible) recipe a primary target, let us instead create a phony recipe on the fly which DEPENDS on all the real recipes. That way, both stamp files as well as tar-balls in PACKAGE_DEPLOY_DIR are taken into account for deciding what needs building.

Aside from drastically reducing the amount of work done, this also seems to make the initialization much faster. For me, the "building dependency tree" time drops from ~26 seconds to ~5 seconds. We also save 6-8% of time/memory on metadata hashing, since we don't compute the metadata for the pseudo-task "build" for all ~300 recipes (only up to and including do_package).

We keep "universe" as a special token to mean [t + ":world" for t in recipe_types].

Villemoes commented 7 years ago

This depends on #208 - I wasn't aware that depending on multiple versions of the same thing wouldn't Just Work(tm).

kimrhh commented 7 years ago

I would really really really like to see this merged soon, as it would lower our waste of cpu cycles significantly