Further automate the server building process

[jefflunt]

• User creation and ssh key provisioning
  • The useradd command, when not specifying a password, might generate the home folder as expected and make the user password-less. With key-based authentication turned on I'd be happy with that http://askubuntu.com/questions/166453/how-to-create-user-account-via-a-bash-script
  • If the user could be provisioned in this manner then it would be trivial to also provision the necessary SSH keys
• What can be done about ENV variable setup?
  • The ~/.bash_aliases file could be a simple copy/paste - not too hard.
  • The ~/.bashrc file and the removal of ~/.bash_profile are probably scriptable
• Configuring the database.yml file
  • RIght now I'm committing a version of the database.yml file that is only an example file. If instead I could load the database settings from the ENV variables and commit this to the repository (so as to keep the secrecy of these values intact) then the copy+past of the ~/.bash_aliases file would take of this step as well. The database.yml file would simply be copied over during a code deploy.
  • A copy of the production vs. staging settings could easily be maintained this way, minimizing the possibility that I would accidentally impact production by mistake
  • Any repo that contains a session secret should move over to the ENV variable method as well

Also a (private) spreadsheet which currently shows that 25% of the server build process is manual. I really need to get this down to something like 1% - the 1% that is the kickoff of the provisioning of the server.

[jefflunt]

Learned how to do some more things with Chef lately. Should be able to provision the user, set the .bashrc and .bash_aliases files easily using templates (including the removal of .bash_profile, and the creation/modification of the database.yml on a per-environment basis.

Should also be able to write a capistrano recipe that will allow me to swap which database a given server points to by recreating the database.yml file by grabbing it from Chef and building it on the fly in capistrano.

Really this provisioning seems a lot simpler now that I've done more with Chef.

[jefflunt]

The swap command should really be something like cap preop deploy:promote

The purpose of this task would be the following:

• SSH into the server, updating the S3QL config, database.yml, and nginx configs to point to production
• Restart the preop server
• Run a smokescreen test
• Move the production IP in EC2 from the current production to the NEW production server
• Stop (halt) the old production server, changing it's label in EC2 to prev.rpglogger.com

This puts the new production server in place, promoting it with all production settings, and rebooting it, giving it a chance to get a clean start. Then, only after the smokescreen test passes will it reassign the IP and stop the old server. There might be some details I'm missing, but this should not only be possible, but relatively easy.

[jefflunt]

There will no longer be a preop server - I will simply rebuild the staging server and promote it when necessary before a big, stack-changing deploy.

[jefflunt]

• The making of the file system could be automated as part of capistrano
• The configuring of the database.yml file could be done via a combination of capistrano calling out to Chef

Once these two things are done, the deploy:install and deploy:setup steps would be fully automated. This means that you could simply do deploy:cold and it should be able to everything in the first two steps, perhaps with a reboot in the middle, and go from zero to deployed app with a single command (after bootstrapping).

I might be able to move all the commands from the bootstrap script into capistrano as a bootstrap recipe, or something similar. It would effectively convert the current script into a series of run "command" commands inside of capistrano, but that's probably 100% okay after all.