Implement installation and deployment tasks in Ansible

@rahulbot & Cindy need a testing server, so we'll have to maintain two servers from now on.

This is a good chance to rewrite our installation and deployment procedures to Ansible.

[x] Install Media Cloud using Ansible
[x] Make handler names lowercase
[x] Install mecab from APT
[x] Try out Docker with Ansible configuration
[x] Make Vagrant use Ansible playbook
[x] Test if Apache has been started after (re)starting it
[x] Test if PostgreSQL has been started after (re)starting it
[x] Deploy code changes using Ansible
[x] Merge apache stop and apache start into a single handler
[x] Test with ansible-lint
[x] Send out Slack notifications when deploying
[x] Make it work against production

Rewrote Media Cloud installation scripts to Ansible in ansible_provisioning branch. The remaining task is to implement automatic deployments using Ansible.

Implemented Ansible roles that:

Set up timezone, locale, hostname, install basic utils
Install Media Cloud system packages (RabbitMQ, Java, Graphviz, MeCab and such)
Install and configure a specific version of PostgreSQL server
Set kernel parameters (ulimit, etc.)
Check out Media Cloud's Git repository if not yet present
Install Python dependencies into Virtualenv
Install Perl dependencies using Perlbrew
Install mecab-ipadic-neologd (needed for Japanese language tokenization)
Copy mediawords.yml.dist to mediawords.yml (if the latter is not yet present)
Set up Git pre-commit hooks
Set up Crontab with all the scripts that we use (Solr imports, new user reports, etc.)
Install and configure Apache 2 to run Media Cloud webapp, optionally with SSL

As per the book, all the rules are idempotent, meaning that Ansible tries to get the host into a specific "state" rather than blindly running a bunch of commands.

Everything runs and gets tested on Travis; test builds are now a little bit slower (because more stuff happens on the test build now), but maybe there's a way to run those builds using Docker on Travis and so speed up the test runs by using an Ubuntu image with pre-installed dependencies, don't know, haven't looked into it yet.

I took the opportunity to simplify the Media Cloud setup a little too:

Python's Virtualenv now gets stored in ~/.virtualenvs/mediacloud/ instead of mc-venv/ so that it can be both 1) used with virtualenvwrapper and 2) be cached on Travis
After some consideration, I've removed Carton and made Ansible install all the Perl dependencies using cpanm to perlbrew-system@mediacloud library under ~/.perlbrew/ because:
- Carton is buggy. I've spent endless hours writing exceptions over exceptions to make it work. Also, quite often one needs to run carton install three times for all the dependencies to get installed.
- Carton is slow. To make the Perl script use dependencies installed with Carton, we have to use run_with_carton.sh wrapper which runs a Perl script that runs another Perl script. Why does it take 500 ms for it to set PERL5LIB is beyond me.
- Carton is hard to maintain. We've both spend yet more hours committing cpanfile.snapshot which always gets overwritten with some new stuff.
- Carton is confusing. Currently we install some Perl dependencies using APT using one version of Perl, some other dependencies "outside of Carton" using another (Perlbrew) version of Perl, and then the majority of dependencies "with Carton" into yet another Perl library.
- One can't use an IDE (e.g. Komodo) with Carton.
- We don't really use Carton's main feature which is dependency version locking. I don't know about you, but whenever I add / remove a Perl dependency, I just commit cpanfile.snapshot and pray for it to work (which it usually doesn't) without giving much thought about which other dependencies got updated. I understand that Carton was introduced to have Perl dependency version locking, but I think that it has since become an outdated nuisance rather than a helpful feature.
If we absolutely positively want to have this dependency version locking (which, I think, we don't really use that much and which is likely obsolete now that we run our unit tests quite often on Travis), I'd much rather have a DarkPAN mirror on S3 and install hardcoded versions of everything from there rather than get back to Carton.
I've merged multiple "initialize some environment variables to use correct Perl / Python and then run something" wrapper scripts into a single run_in_env.sh. The script initializes Perlbrew environment, Virtualenv environment, and then runs whatever command was passed as its argument. This will allow us to have a single wrapper for running both Perl and Python scripts.

For example, with run_with_carton.sh one has to run a single unit test as follows:
```
# Call `carton exec`, which will run `prove`, which
# requires an additional Perl include path
./script/run_carton.sh exec prove -Ilib/ unit_test.t
```
With the run_in_env.sh script, one can do:
```
# Initialize Perlbrew, add "lib/" to Perl
# include path, initialize Virtualenv,
# run the argument command
./script/run_in_env.sh prove unit_test.t
```
...or you can simply start a shell and run perl / python normally:
```
$ ./script/run_in_env.sh bash
mediacloud$ prove unit_test.t
```
I hope this will save some development time for us.

This is terrific. Very exciting for us to move forward with a more sane deployment system. The better environment system looks great as well.

How do we track CPAN dependencies now?

As I think you sense, I'm nervous about losing the version locking. The current system was not locking perfectly, but it was greatly reducing the amount of constant version shifting to the relatively rare times when we added a new module to cpanfile. Without that protection, we will likely be getting frequent (daily?) version changes coming down the stream from CPAN. The carton stuff was not added just for theoretical safety -- it was added because we were having frequent problems with module upgrades breaking things in production and with new install not working.

As you point out, we now have the protection of the daily vagrant test to help us find dependency tests as they arise, so hopefully that will be enough to keep things sane. We also have the protection of CPAN being much less active these days, so we'll just get fewer upgrades coming through the pipe.

There's also just an inherent danger in the developers running different versions of modules on their system than what is running in production (and in various production systems running different versions of modules). We could mitigate that problem by adding a module update process to the deployment that insures that we have the latest version of all modules every time we deploy (and also run this before we test every time). That way we're at least consistent across all of our dev, testing, and production systems.

-hal

On Thu, Aug 17, 2017 at 7:16 PM, Linas Valiukas notifications@github.com wrote:

Rewrote Media Cloud installation scripts to Ansible in ansible_provisioning https://github.com/berkmancenter/mediacloud/tree/ansible_provisioning branch. The remaining task is to implement automatic deployments using Ansible.

Implemented Ansible roles that:

Set up timezone, locale, hostname, install basic utils

Install Media Cloud system packages (RabbitMQ, Java, Graphviz, MeCab and such)

Install and configure https://github.com/berkmancenter/mediacloud/tree/ansible_provisioning/ansible/roles/postgresql-server/files/postgresql_conf a specific version of PostgreSQL server

Set kernel parameters (ulimit, etc.)

Check out Media Cloud's Git repository if not yet present

Install Python dependencies into Virtualenv

Install Perl dependencies using Perlbrew

Install mecab-ipadic-neologd (needed for Japanese language tokenization)

Copy mediawords.yml.dist to mediawords.yml (if the latter is not yet present)

Set up Git pre-commit hooks

Set up Crontab with all the scripts that we use (Solr imports, new user reports, etc.)

Install and configure https://github.com/berkmancenter/mediacloud/blob/ansible_provisioning/ansible/roles/apache2-fcgi/templates/001-mediacloud.conf.j2 Apache 2 to run Media Cloud webapp, optionally with SSL

As per the book, all the rules are idempotent, meaning that Ansible tries to get the host into a specific "state" rather than blindly running a bunch of commands.

Everything runs and gets tested on Travis; test builds are now a little bit slower (because more stuff happens on the test build now), but maybe there's a way to run those builds using Docker on Travis and so speed up the test runs by using an Ubuntu image with pre-installed dependencies, don't know, haven't looked into it yet.

I took the opportunity to simplify the Media Cloud setup a little too:

-

Python's Virtualenv now gets stored in ~/.virtualenvs/mediacloud/ instead of mc-venv/ so that it can be both 1) used with virtualenvwrapper https://virtualenvwrapper.readthedocs.io/en/latest/ and 2) be cached on Travis

After some consideration, I've removed Carton and made Ansible install all the Perl dependencies using cpanm to perlbrew-system@mediacloud library under ~/.perlbrew/ because:

Carton is buggy. I've spent endless hours writing exceptions over exceptions https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_with_carton.sh to make it work. Also, quite often one needs to run carton install three times for all the dependencies to get installed.

Carton is slow. To make the Perl script use dependencies installed with Carton, we have to use run_with_carton.sh wrapper which runs a Perl script that runs another Perl script. Why does it take 500 ms for it to set PERL5LIB is beyond me.

Carton is hard to maintain. We've both spend yet more hours committing cpanfile.snapshot which always gets overwritten with some new stuff.

Carton is confusing. Currently we install some Perl dependencies using APT using one version of Perl, some other dependencies "outside of Carton" https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_outside_of_carton.sh using another (Perlbrew) version of Perl, and then the majority of dependencies "with Carton" https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_with_carton.sh into yet another Perl library.

One can't use an IDE (e.g. Komodo https://www.activestate.com/komodo-ide) with Carton.

We don't really use Carton's main feature which is dependency version locking. I don't know about you, but whenever I add / remove a Perl dependency, I just commit cpanfile.snapshot and pray for it to work (which it usually doesn't) without giving much thought about which other dependencies got updated. I understand that Carton was introduced to have Perl dependency version locking, but I think that it has since become an outdated nuisance rather than a helpful feature.

If we absolutely positively want to have this dependency version locking (which, I think, we don't really use that much and which is likely obsolete now that we run our unit tests quite often on Travis), I'd much rather have a DarkPAN mirror on S3 http://blogs.perl.org/users/steven_haryanto/2014/01/installing-modules-from-cpan-and-your-own-darkpan.html and install hardcoded versions of everything from there rather than get back to Carton.

I've merged multiple "initialize some environment variables to use correct Perl / Python and then run something" wrapper scripts into a single run_in_env.sh. The script initializes Perlbrew environment, Virtualenv environment, and then runs whatever command was passed as its argument. This will allow us to have a single wrapper for running both Perl and Python scripts.

For example, with run_with_carton.sh one has to run a single unit test as follows:

Call carton exec, which will run prove, which

requires an additional Perl include path

./script/run_carton.sh exec prove -Ilib/ unit_test.t

With the run_in_env.sh script, one can do:

Initialize Perlbrew, add "lib/" to Perl

include path, initialize Virtualenv,

run the argument command

./script/run_in_env.sh prove unit_test.t

...or you can simply start a shell and run perl / python normally:

$ ./script/run_in_env.sh bash mediacloud$ prove unit_test.t

I hope this will save some development time for us.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/berkmancenter/mediacloud/issues/179#issuecomment-323227159, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvvTx-xHUi2_tQ8b-WzXdMv_ZRxAopvks5sZNfrgaJpZM4OdgN4 .

-- Hal Roberts Fellow Berkman Klein Center for Internet & Society Harvard University

This is a pain on the front end as well, as it pertains to Javascript npm dependencies. Things were just changing too fast for us to not specific module versions. So we have our version lock file checked in and upgrade manually every month or two (we track this on a bug). Some modules are purposefully held back to older versions because newer ones would require extensive changes that don't give us much of a win. That's the world of JS right now...

Sorry for not being clear. I'm like dependency version locking, I just don't like the tool that we use for it (Carton) due to things listed in the above comment, and so I would like to replace it with some other option.

As far as I see it, the options are:

Don't do any dependency version locking. Rey on the fact that there isn't much action on CPAN these days, and of something changes significantly, the testing is likely to catch it. Pros: easy to do, already works. Cons: technically unsafe.
Hardcode versions of direct dependencies, don't worry about the sub-dependencies that cpanm installs. We only care about interfaces of modules that we use directly and not so much about the modules (and their versions) that are used by those dependencies. Pros: still easy to do, slightly safer than doing nothing. Cons: not 100% safe, but then the testing should be able to catch those rare occurrences when a sub-sub-sub-dependency breaks?
Maintain DarkCPAN mirror with a copy of modules and specific versions of them that we use. I've made one (https://github.com/berkmancenter/mediacloud/commit/096f41bc59f9063b4308ccf7a24f5bb7bc65c897) and it seems to work fine. Pros: super paranoid. Cons: while downloading all the CPAN modules and publishing them to S3 could be done with a simple script, maintaining such a mirror still involves some semi-manual work; adding a new CPAN dependency is no longer that trivial.
Try to live with Carton and continue with semi-locked dependencies that we are able to install with the tool. Pros: somewhat tried and tested way. Cons: very hard to maintain, a big share of dependencies have to be installed separately with cpanm and without any version locking, massive cpanfile.snapshot file with constant commits to it.

I'd go with 2 or 3.

Some more developments:

After trying out various options for installing Perl dependencies and locking their versions, I've opted for snapshotting the whole CPAN repository to S3 and installing Perl modules from it, i.e. maintaining a MiniCPAN repository of the latest versions of all the modules. This is a bit of a shotgun approach, but:
- Adding new dependencies is (still) easy, one has to add them to this array.
- Installing modules with cpanm works each and every time without having to cherry-pick which module depend on other modules.
- Updating the repository is relatively easy too, you just run a script and wait for 30 minutes for it to fetch 4 GB of latest CPAN modules and upload them to S3.
I hope this will be a better solution than Carton was, work out for us in the long term and wouldn't become a maintenance headache.
To make life with Travis easier and have a chance to try out Docker, I've build a Docker image based on ubuntu:16.04 and pre-installed with Media Cloud dependencies (system, Python and Perl ones). Travis then fetches the image, creates a test container out of it, copies the code to be tested to that container, runs the full Ansible script to update the container with the latest dependencies, and lastly executes the test suite on the container. As a result:
- We'll no longer need to support 12.04 / 14.04 to make Media Cloud testable on Travis.
- Builds are to be more predictable as now we can see what's failing on our own local machines using the very same Docker image.
- Builds are slightly faster too because now only ~3:30 minutes are being spend on preparing the environment for the build (fetching the Docker image with pre-installed dependencies and such).

Main remaining task is to automatically deploy code and restart services using Ansible.

(Clicked "Comment" too early on previous comment, updated it since.)

Ansible branch merged and deployed. The summary of this Brave New World:

There's a short README describing the Ansible setup.
To set everything up on your local machine without much hassle, run ./install.sh (which, in turn, will run the Ansible script).
Perl dependencies:
- There's no more Carton (and no local/ with dependencies), everything gets installed into Perlbrew's perl-system@mediacloud library under ~/.perlbrew/libs/perl-system@mediacloud/.
- Perl dependencies are now listed in an Ansible variable under ansible/roles/perl-dependencies/vars/main.yml.
- Perl dependencies are locked by installing them from our own MiniCPAN repository on S3. That MiniCPAN contains all the CPAN packages that were available for download on 2017-08-24. There's a dependency update script tools/cpan/mirror-cpan-on-s3.sh available that will fetch all the modules from CPAN and upload them to S3 again (takes about 30 minutes and 4 GB on a good connection).
- We no longer need to have the FindLib shim at the top of every script as run_in_env.sh sets that path automatically.
Python dependencies:
- Python's virtualenv moved from mc-venv/ to ~/.virtualenvs/mediacloud/ to make it compatible with virtualenvwrapper (should you choose to use it).
- Python dependencies are not listed in an Ansible variable under ansible/roles/python-dependencies/vars/main.yml.

run_with_carton.sh, etc. has been merged into a single run_in_env.sh script which initializes both Perl and Python environments correctly:

# Run a Perl script (with a correct '#!/usr/bin/env perl' shebang)
$ ./script/run_in_env.sh perl_script.pl

# Run a Python script (with a correct '#!/usr/bin/env python' shebang)
$ ./script/run_in_env.sh python_script.py

# Run Bash, use it to run anything you want
$ ./script/run_in_env.sh bash
mediacloud$ ./perl_script.pl
mediacloud$ ./python_script.pl
mediacloud$ exit

I'll send you a sample Ansible inventory file that you can use for deployments against our production in a private email.
Travis now pulls dockermediacloud/mediacloud-travis with Ubuntu 16.04 and Media Cloud system / Perl / Python dependencies preinstalled, reruns the full Ansible setup script and then continues with the tests. Sometimes we might need to rebuild that test image using ansible/Dockerfile to speed things up a little bit, but that's not going to happen too often. Travis run time is about the same.
It is (or should be) completely safe to rerun both setup.yml and deploy.yml playbooks multiple times against production.
There's a Slack channel #deployments to which Ansible will send notifications about the code being deployed.
It comes without saying, but if you have a new Cron job to set up, PostgreSQL setting to toggle, Perl / Python dependency to install, Apache configuration property to adjust, etc., please set it up in an Ansible script instead of editing things on production manually.

Let me know if something doesn't work or if you have questions.

mediacloud / backend

Implement installation and deployment tasks in Ansible #179

Python's Virtualenv now gets stored in ~/.virtualenvs/mediacloud/ instead of mc-venv/ so that it can be both 1) used with virtualenvwrapper https://virtualenvwrapper.readthedocs.io/en/latest/ and 2) be cached on Travis

Call `carton exec`, which will run `prove`, which

requires an additional Perl include path

Initialize Perlbrew, add "lib/" to Perl

include path, initialize Virtualenv,

run the argument command