search

GetPoplog / Seed

Scripts for getting Poplog onto your Linux machine
MIT License
7 stars 3 forks source link

Split Makefile into separate parts #116

Closed willprice closed 3 years ago

willprice commented 3 years ago

This PR breaks up the Makefile into a bunch of fragments that are include-ed into the main Makefile. I'm not 100% sure on this change. On one hand it makes it feel a bit more manageable to split the file up into orthogonal parts. One the other hand, there are dependencies across files, which is perhaps a little counter intuitive.

For reference, the semantics of include are to

  1. pause the reading of the current Makefile.
  2. process the referenced Makefile until completion.
  3. switch back to continuing to process the original Makefile.

(see https://www.gnu.org/software/make/manual/html_node/Include.html for more details on include)

willprice commented 3 years ago

My grand aim is to split the build process into two steps, managed by different makefiles.

  1. The main Makefile, will include all the helper targets, packaging etc
  2. The build Makefile, which will be copied into the _build folder along with all the necessary source files so that proper dependencies can be set up.

Currently proper dependency tracking is (almost) impossible since our current Makefile downloads and unzips directories as part of the build process. This means it is impossible to use $(wildcard *.ext) to match files that are produced as part of the download. Technically it is possible to circumvent this issue by hard coding all the files that will be produced as a result of unzipping the files, however this would be rather unpleasant to manage (and would require some Makefile generation machinery to manage in a non-brittle manner).

Instead, I'm proposing (to be completed in a follow up PR) to assemble the skeleton of the _build directory with all the necessary files (e.g. unzipped packages, base files) which will then enable the writing of a sane Makefile that can make use of wildcard matches to set up deps properly.

One issue I envisage with this approach is in managing the dependencies of base correctly. Ideally we'd have it so that only the necessary build steps are re-executed when a single file in base is modified. However, this is made challenging if we copy of base to _build/poplog_base in one go. As an alternative we could copy each file individually with proper provenance tracking, e.g.

BASE_FILES:=$(shell cd base && find . -type f -printf '%P\n')
BASE_TARGET:=$(BASE_FILES:%=_build/poplog_base/%)

$(BASE_TARGET): _build/poplog_base/%: base/%
    mkdir $(@D)
    cp $< $@

However, this is understandably rather slow due to the many thousands of cp invocations that are spawned.

But without tracking the dependencies on base, a lot of the benefits of a proper build system are no longer possible. Instead, we'd just end up copying over all of base again, triggering a rebuild of much of the system... unless the mtime of the base files are maintained when copying the files into the _build/poplog_base directory or if rsync or similar is used to only copy the updated files. This is something I've only just thought of whilst writing this, so I don't know whether it'd work or not.