Modifying deployment configs per-host

[auspex]

Under 1.0a3, I created a hostout configuration for deploying to 8 hosts (actually, most are on the same physical machine, but they don't have to be), which involved a buildout.cfg:

...
[hostout]
host   = xxx.ca
user   = plone
path   = /home/plone4/plone-${:_buildout_section_name_}
parts  =  
    filestorage
    zeoserver
    instance
    zopepy
    crontab_zeopack

buildout = deploy.cfg
pre-commands  = /home/plone4/bin/supervisorctl stop  ${:_buildout_section_name_}-plone     
post-commands = /home/plone4/bin/supervisorctl start ${:_buildout_section_name_}-plone     
buildout-cache = /home/plone4
effective-user = plone
buildout-user  = plone
user           = derek
identity-file  = /home/derek/.ssh/id_rsa

[controller]
<=hostout
recipe = collective.hostout
parts  = supervisor
extends= pointerstop.recipe.hostout 
buildout = 
path   = /home/plone4
pre-commands  = 
post-commands = 
include = 
    ApacheRewrite.conf

[cmb]
recipe = collective.hostout
<= hostout
zeo-storage = CMB
...

and for each of the "hosts" (e.g., 'cmb' above) I could add parts to the corresponding .cfg file (cmb.cfg):

[buildout]
newest = true
eggs-directory = /home/plone4/eggs
versions = versions
develop = 
parts = filestorage zeoserver instance zopepy
extends = hostoutversions.cfg cmb_base.cfg
download-cache = /home/plone4/downloads

[versions]
...

[filestorage]
parts = cmb
recipe = collective.recipe.filestorage

[filestorage_cmb]
zeo-storage = CMB

[instance]
port-base = 0

[zeoserver]
zeo-address = 12005

Here, the [buildout] and [versions] sections are controlled by the collective.hostout configuration, but the remainder are my specific configurations for that particular host. In fact, all of these (except port-base in [instance] are updated by the pointerstop.recipe.hostout plugin in the [controller] part. Additionally, the plugin writes the ApacheRewrite.Conf file (which puts the appropriate proxy rules for each host into the Apache configuration), and the supervisord config file allowing one supervisord daemon to monitor all the Plones. The "port-base" setting is what determines the variant part of what gets writtent to those two config files.

In 1.0a5, you no longer write "cmb.cfg". Instead, you write a new cmb-VERSION.cfg every time, and this has none of my configuration.

The simplest solution to get back my previous functionality is to make "cmb-VERSION.cfg" always default to the contents of "cmb.cfg" (which of course may not exist, and is therefore the same as using an empty file), but if there's a way to have different configs deployed for each host without further modification of collective.hostout, I'd be even happier.

[auspex]

If we want to just recover backwards-compatibility to 1.0a3, I've commited a change to my fork, and I'll send a pull request that you can just reject if we have a better solution.

[djay]

I don't think going back to generating files locally is the way to go. It was problematic in that it could overwrite files named the same causing strange errors. Rewriting generated buildout files doesn't seem that great an idea. it would be better to make this part of the actual hostout process.

I think instead the following options are open. a) just write your own top level buildout files for each deployment. Thats what I do. b) we put in some kind of syntax for inserting parts and attributes into hostout buildout spec. This is kind of buildout within buildout though and I can't think of an elegant way to do this nicely yet. c) uses fabfiles= option to specify a fabfile which can plugin into the hostout process and adjust the buildout file generation so it can be overridden by this fabfile.

[auspex]

On Mon, Apr 25, 2011 at 4:30 AM, djay < reply@reply.github.com>wrote:

I don't think going back to generating files locally is the way to go. It was problematic in that it could overwrite files named the same causing strange errors. Rewriting generated buildout files doesn't seem that great an idea. it would be better to make this part of the actual hostout process.

I don't see the problem. You do generate a buildout file - I don't say rewrite it (and my patch doesn't do that), all I'm suggesting is that you have a way to provide defaults for that file. And you call it "problematic"

• I call it a tool! I've been relying on it overwriting that file.

I think instead the following options are open. a) just write your own top level buildout files for each deployment. Thats what I do.

How can that work? [never mind - see next comment for solution]

b) we put in some kind of syntax for inserting parts and attributes into hostout buildout spec. This is kind of buildout within buildout though and I can't think of an elegant way to do this nicely yet.

The very least I need is a variable containing the name of the host. I can do a lot with that.

c) uses fabfiles= option to specify a fabfile which can plugin into the hostout process and adjust the buildout file generation so it can be overridden by this fabfile.

To do this on a per-host basis doesn't handle my use-case. I have one plugin - invoked in one part - that updates configuration for all the others. I don't have to do it that way to update my Apache rewrite rules (1 line per host), or to update the individual host configs, but I think I do have to do it that way to update the supervisord config which needs a value

for each of the programs and eventlisteners variables.

derek

[auspex]

D'oh. Using a per-host config is trivially easy!

[hostout]
buildout = ${:_buildout_section_name_}.cfg
...
[cmb]
recipe = collective.hostout
<= hostout

It would be nice, however, to be able to add arbritrary variables to the buildout section. Perhaps a "buildout-additional" variable like:

buildout-additional =
    var1 = value
    var2 = value

[djay]

I guess that wouldn't be too bad. But you can do most things fairly cleanly just with using multiple buildout files in the hostout section e.g.

[web1dev] <= hostout buildout = base.cfg web.cfg 1.cfg