brettviren / worch

Let the orchestration waf through the suite.
5 stars 3 forks source link

Removal of temporaries #43

Open brettviren opened 10 years ago

brettviren commented 10 years ago

The current build of "art" produces 17GB in the installation directory (as reported by Ben Morgan). The way that this build relies on Fermilab build scripts means that most of this is due to temporary files produced during their run and are not strictly something that should be installed.

For this case, we should see if Fermilab supplies a cleaning mechanism.

In a worch build where worch is allowed to control the location of the temporaries doing an rm -rf tmp/ from the worch/ directory is enough to remove all temporaries.

Regardless of how they are removed, some thought is needed in how to support this removal in worch if one expects to reuse the current installation when building a new release.

This has not been tested yet (by me) but I suspect waf may rebuild temporaries if a new configuration is used even if it happens to reuse some identical package-level configuration. Finding ways to minimize rebuilding with and w/out some level of purging of temporaries would be good.

As sites accrue more and more releases this will become more important.

brettviren commented 10 years ago

The files under tmp/controls/ that are created on each successful step and which are a large ingredient in the step dependency graph will need special attention in any purge scheme.

When a new suite is built in place of a prior one which has been purged (in some way) it is desirable to not rebuild any common packages that have already been built. To support this the "control" files will probably need gain more information such as a version string.

More thought would also be needed if the same build area will be used for different platforms or qualifiers (to take the Fermilab term).

The complexity this would add may not be worth the time savings.

brettviren commented 10 years ago

Another approach might be to allow more than one version of any given package to be in a "suite" and then treat the "suite" as the description of the entire installation base. As new releases are produced their configuration is folded in and as releases get purged its unique chunks of configuration get extricated. How to do this in a sane manner needs some good ideas.