Write script to automate deployment

[briancroxall]

Right now I know @rlskoeser is having to manually push the updates from dev to production. We need to get some scripts that could automate this process. In particular, I think that the CHNM server admins might need to help us with this. @rsanc77 and @patrickmj?

[rlskoeser]

I did some quick looking today and it seems that fabric ( http://docs.fabfile.org/en/1.7/ ) is still one of the better options for a deploy script for django, which is what we use at Emory, so I'm familiar with it and could probably write one fairly easily based on some of our existing fab scripts here. I will need to confer with a sysadmin type person for some of the setup and permissions details, and also need to know how much you are comfortable with us automating. We'll also probably need a dev/test system configured as closely as possible to the production (at least for a little while) so we can test the deploy script without breaking anything.

Ideally, once the script is written and user accounts are configured appropriately on the server, any one of a small number of team members could run the script to update the site. I should be able to build in a simple rollback mechanism as well, to revert to the previously deployed version of something went wrong.

If there are any best practices or things you all are using that are working well for deploying and monitoring any of the other CHNM software tools & sites, I'd like to hear about them in case there is anything we can adapt or apply here.

[briancroxall]

Is this something that Jay could help us troubleshoot, @rlskoeser?

[rlskoeser]

@briancroxall - seems like a good idea to at least consult with him on the best way to set things up

[rsanc77]

We currently use capistrano for omeka.net deployment but it does require a specific file structure. I like fabric more because it does have that flexibility when working with existing projects.

[rlskoeser]

Here's an example fabfile from one of our Emory Libraries projects: https://github.com/emory-libraries/findingaids/blob/master/fabfile.py

Basically, it builds a local copy of the current version of the code, tars it up and copies up to the server, untars within a specific directory structure, sets up a brand new python virtualenv, copies in a localsettings file already on the server (sensitive configs not checked into git) from a known location, runs django collectstatic to prep all the image/javascript/css files to be served out. Our process isn't completely automated, but I think we could tweak it to get there-- e.g., apache conf file could be tweaked by the fab script and just included / symlinked in the main apache config (we may need to clean up / address some issues/quirks in the current config before we can do this properly).

The directory structure we use looks somewhat like this:

• top level deploy directory is /home/httpd/sites/appname/ (but base path could be anything)
• particular deployed version goes under /home/httpd/sites/appname/appname-version-0.1.1
• unversioned settings go at the top level
• fab file generates/updates symlinks for current & previous so we can easily keep track
• actual running apache path could use the symlink so it auto-updates on deploy + apache restart

I'd like it set up so there is an account for the app, and that account owns all the deploy files, and then everyone who has access to deploy would have sudo access to the app account. If we are going for automating it fully, we'd have to have sudo access to restarting apache (which I think you already gave me).

I can also set up the fabric deploy so it will clean up old versions of the deploy (e.g., only leave the X oldest deploys and remove the rest).

The one part I'm not sure about automating is the database part - if we get to the point where we're modifying the database with our updates (which could be the case), django has scripts for migrating the db, and depending on the route we go it can handle the migrate up & down for us, but it would be good if there is an easy way to snapshot and restore the database if we do have to revert things for some reason.

I'm not too tied to any of what I'm proposing here, this is just what I know and could easily script based on our other deploy scripts.

[rsanc77]

Sounds good to me I'll create the user and give him ownership of the app. With respect to the database it would be ideal to use the following mysqldump -u username -p databaseToSave | gzip -9 /path/to/stored/mysqlfiles/restorationpoint.sql.gz which should be run prior to or during the archiving process and store it with the archive or in a folder of mysqlbackups. That way you have a copy of your database just in case you need it. If you need to restore you can just use the following command gunzip < /path/to/stored/mysqlfiles/restprationpoint.sql.gz | mysql -u username -p databaseToRestore.

The only problem here is if the database becomes very large. If that happens mysqldump might delay the deployment process. I think this will make migrating up or down easy.

any preference on the name for the new user?

[rlskoeser]

Thanks for the mysql dump/restore commands. It looks like it should be pretty straightforward to add fab tasks for that. In fact, now that I think more about it, we may be able to take advantage of some django built-in manage commands-- which would have easy access to database configuration & username stuff.

The database isn't very large right now (we're basically only storing session data), so maybe this is just something to keep an eye on as the app expands. A couple of features we're looking would probably have db components, but I'm not sure there's anything terribly substantial.

As for the username - I've been using serendip as a short-form of the app name. Work for you? @briancroxall any thoughts / do you care?

Oh, also - I think at some point we'll need a test server configured exactly like production so we can test the deploy process without messing up production until we get all the details squared away. (I can't figure out how else we could reliably test it - open to suggestions.) This can be as temporary and low-powered and cheap as you want/need it to be (e.g. minimal AWS server we only turn on when we're ready to test the deploy?), all I care is that the environment is consistent.

[mialondon]

I second that - it won't solve our branch/master requirement but it would solve the need for a QA server and reduce fears about the differences between the AWS and Heroku environments.

[briancroxall]

@rlskoeser I think serendip works well for the user name and it's short enough that it's actually type-able, which is not really the case with Serendip-o-matic.

I also agree that the proposal for a test server for QA is important. Rebecca's ideas for it being something that we can spin up and down for short-term testing is ideal.

[rsanc77]

I'll set up the serendip user. I will have to get back to you on spinning another ec2-instance.

[rlskoeser]

@rsanc77 - doesn't have to be ec2. It could be a repurposed desktop set up as a temporary server for all I care! I don't know what your options are and it sounded like you couldn't upgrade mod wsgi/mod python to 2.7 on the chnm dev server. I just need a test environment where we can be reasonably certain that configuring apache, installing python libraries, and dumping & restoring the database will behave the same way it does in production.

[rsanc77]

The user serendip is created and has ownership of the serendipomatic folder. @rlskoeser you can find a tar file with the user information in your home folder. swilliams, @rlskoeser and awilliams have sudo privileges and can restart the apache server. When doing so please use $service httpd start|stop|restart|configtest... not /etc/init.d/httpd. Unfortunately I couldn't update mod_wsgi sorry. But you can easily test it by spinning up a virtual machine with centos 6.x which has the same version of python that the ec2 instance uses. I prefer virtualbox. Do you need the user serendip to have the ability to restart apache as well?

[rlskoeser]

If our user accounts have sudo access to restart apache then I don't think the serendip account needs it.

Virtualbox sounds like a great idea. Would it be possible/not too much trouble for you to create a virtualbox image that matches the production environment? If you can create an image I should be able to run a copy locally for testing.

[rsanc77]

I place the virtualbox image in the home directory of the production server. it has the following users. Myself, @rlskoeser and serendip.

The credentials for all the users are the same in the image as they are in the production server. @rlskoeser you have full admin privileges.

file structure is the same for the application and log files in both the production and virtualbox image. Make sure to add the ip address of the image to your /etc/hosts file that way you can view any changes. If you encounter any issues or have any problems using virtualbox let me know and I will try to help

[rlskoeser]

@rsanc77 : I downloaded the virtualbox appliance and was able to get it imported and running on my macbook. If you have any tips about configuring the networking so I can ssh from the host machine to the guest virtualbox, let me know. Otherwise it looks like I will have to do some more digging.

Once I get the virtualbox networking figured out, I should be all set to start adapting and testing a fab deploy script.

[rlskoeser]

If I set the network to use 'bridged adapter' and then login and run ifconfig I can see the IP address the virtual server is getting via DHCP. Works on my home network, at least. (Can't remember if I tried this at work or not.) Once I have the IP address, it looks like I can basically treat it the same way I would the remote server. Should be all set to start working on the actual deploy script.

[rlskoeser]

@rsanc77 - couple questions for you. I finally had a chance to look at this, & think I have it mostly working on the virtualbox server (with notes on the adjustments that will need to be made on the live server). Hoping you can help with a couple small details relating to the sql dump commands.

The sql dump/gzip command you gave me didn't quite work; I ended up with this (seem right?):

mysqldump -u username --password=### dbname | gzip > /tmp/dbname.sql.gz

I haven't tried the restore command yet; can that be done on top of an existing database?

The main thing I'm struggling with is, I'm not sure how to pass/where to specify the db credentials. They aren't readily available on the local workstation where I run the fab deploy script; they are in the django settings on the remote server where the webapp runs, but getting them out usefully seems tricky. I was wondering if you thought it would be reasonable for me to create two simple shell scripts on the server, which would include the db credentials.

Other detail questions:

• any recommendations for where the database backup should be put? (does it matter?)
• would you thinking of keeping multiple versioned backups, or just a single backup that can be restored by the deploy script if (and only if) we need to revert to the previous version?

FYI, the database is currently very small, so generating the backup is quite quick.