search

mattray / spiceweasel

Generates Chef knife commands from a simple JSON or YAML file.
Apache License 2.0
284 stars 65 forks source link

Request: Bulk node creation, allow for asynchronous layer creation? #73

Open Igneous opened 10 years ago

Igneous commented 10 years ago

It would be really great if there were a way to specify a layers in a bulk-provision environment that can come up asynchronously.

clusters:
- drupal:
  - ec2 1:
      run_list: role[drupal_lb]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{{n}} -S bwolfe
  - ec2 1:
      run_list: role[drupal_db]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-db-{{n}} -S bwolfe
  - ec2 2:
      run_list: role[drupal_elasticsearch]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-es-{{n}} -S bwolfe
  - ec2 3:
      run_list: role[drupal_web]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-web-{{n}} -S bwolfe

In this example, after the load balancer and database servers are up, the web servers and the elasticsearch boxes can safely come up in parallel, It would be nice to be able to have spiceweasel generate those commands so they're executed at the same time.

This idea could be represented like this (notice the & on ec2 2):

clusters:
- drupal:
  - ec2 1:
      run_list: role[drupal_lb]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{{n}} -S bwolfe
  - ec2 1:
      run_list: role[drupal_db]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-db-{{n}} -S bwolfe
  - ec2 2&:
      run_list: role[drupal_elasticsearch]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-es-{{n}} -S bwolfe
  - ec2 3:
      run_list: role[drupal_web]
      options: -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c
        -g SNIP --subnet SNIP -N bwolfe-drupal-web-{{n}} -S bwolfe

That would generate commands similar to:

seq 1 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-lb-{} -S bwolfe -E drupal -r 'role[drupal_lb]'
seq 1 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-db-{} -S bwolfe -E drupal -r 'role[drupal_db]'
seq 2 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-nagios-test-{} -S bwolfe -E drupal -r 'role[drupal_elasticsearch]' &
seq 3 | parallel -u -j 0 -v -- knife ec2 server create -i ~/bwolfe.pem -x ubuntu -I SNIP --region us-east-1 -Z us-east-1c -g SNIP --subnet SNIP -N bwolfe-drupal-web-{} -S bwolfe -E drupal -r 'role[drupal_web]'

(Note the presence of & at the end of line 3, effectively executing the knife create commands for both drupal_elasticsearch boxes and drupal_web boxes in parallel)

mattray commented 10 years ago

Thanks for the feedback. My current plan is to make Spiceweasel 3.0 (the next, next release) become a front-end for generating Chef Metal recipes. Metal is working on support for parallel and asynchronous creation, which should negate the usage of GNU parallel completely.