Closed pypt closed 6 years ago
Rewrote Media Cloud installation scripts to Ansible in ansible_provisioning
branch. The remaining task is to implement automatic deployments using Ansible.
Implemented Ansible roles that:
mediawords.yml.dist
to mediawords.yml
(if the latter is not yet present)As per the book, all the rules are idempotent, meaning that Ansible tries to get the host into a specific "state" rather than blindly running a bunch of commands.
Everything runs and gets tested on Travis; test builds are now a little bit slower (because more stuff happens on the test build now), but maybe there's a way to run those builds using Docker on Travis and so speed up the test runs by using an Ubuntu image with pre-installed dependencies, don't know, haven't looked into it yet.
I took the opportunity to simplify the Media Cloud setup a little too:
Python's Virtualenv now gets stored in ~/.virtualenvs/mediacloud/
instead of mc-venv/
so that it can be both 1) used with virtualenvwrapper and 2) be cached on Travis
After some consideration, I've removed Carton and made Ansible install all the Perl dependencies using cpanm
to perlbrew-system@mediacloud
library under ~/.perlbrew/
because:
carton install
three times for all the dependencies to get installed.run_with_carton.sh
wrapper which runs a Perl script that runs another Perl script. Why does it take 500 ms for it to set PERL5LIB
is beyond me.cpanfile.snapshot
which always gets overwritten with some new stuff.cpanfile.snapshot
and pray for it to work (which it usually doesn't) without giving much thought about which other dependencies got updated. I understand that Carton was introduced to have Perl dependency version locking, but I think that it has since become an outdated nuisance rather than a helpful feature.If we absolutely positively want to have this dependency version locking (which, I think, we don't really use that much and which is likely obsolete now that we run our unit tests quite often on Travis), I'd much rather have a DarkPAN mirror on S3 and install hardcoded versions of everything from there rather than get back to Carton.
I've merged multiple "initialize some environment variables to use correct Perl / Python and then run something" wrapper scripts into a single run_in_env.sh
. The script initializes Perlbrew environment, Virtualenv environment, and then runs whatever command was passed as its argument. This will allow us to have a single wrapper for running both Perl and Python scripts.
For example, with run_with_carton.sh
one has to run a single unit test as follows:
# Call `carton exec`, which will run `prove`, which
# requires an additional Perl include path
./script/run_carton.sh exec prove -Ilib/ unit_test.t
With the run_in_env.sh
script, one can do:
# Initialize Perlbrew, add "lib/" to Perl
# include path, initialize Virtualenv,
# run the argument command
./script/run_in_env.sh prove unit_test.t
...or you can simply start a shell and run perl
/ python
normally:
$ ./script/run_in_env.sh bash
mediacloud$ prove unit_test.t
I hope this will save some development time for us.
This is terrific. Very exciting for us to move forward with a more sane deployment system. The better environment system looks great as well.
How do we track CPAN dependencies now?
As I think you sense, I'm nervous about losing the version locking. The current system was not locking perfectly, but it was greatly reducing the amount of constant version shifting to the relatively rare times when we added a new module to cpanfile. Without that protection, we will likely be getting frequent (daily?) version changes coming down the stream from CPAN. The carton stuff was not added just for theoretical safety -- it was added because we were having frequent problems with module upgrades breaking things in production and with new install not working.
As you point out, we now have the protection of the daily vagrant test to help us find dependency tests as they arise, so hopefully that will be enough to keep things sane. We also have the protection of CPAN being much less active these days, so we'll just get fewer upgrades coming through the pipe.
There's also just an inherent danger in the developers running different versions of modules on their system than what is running in production (and in various production systems running different versions of modules). We could mitigate that problem by adding a module update process to the deployment that insures that we have the latest version of all modules every time we deploy (and also run this before we test every time). That way we're at least consistent across all of our dev, testing, and production systems.
-hal
On Thu, Aug 17, 2017 at 7:16 PM, Linas Valiukas notifications@github.com wrote:
Rewrote Media Cloud installation scripts to Ansible in ansible_provisioning https://github.com/berkmancenter/mediacloud/tree/ansible_provisioning branch. The remaining task is to implement automatic deployments using Ansible.
Implemented Ansible roles that:
- Set up timezone, locale, hostname, install basic utils
- Install Media Cloud system packages (RabbitMQ, Java, Graphviz, MeCab and such)
- Install and configure https://github.com/berkmancenter/mediacloud/tree/ansible_provisioning/ansible/roles/postgresql-server/files/postgresql_conf a specific version of PostgreSQL server
- Set kernel parameters (ulimit, etc.)
- Check out Media Cloud's Git repository if not yet present
- Install Python dependencies into Virtualenv
- Install Perl dependencies using Perlbrew
- Install mecab-ipadic-neologd (needed for Japanese language tokenization)
- Copy mediawords.yml.dist to mediawords.yml (if the latter is not yet present)
- Set up Git pre-commit hooks
- Set up Crontab with all the scripts that we use (Solr imports, new user reports, etc.)
- Install and configure https://github.com/berkmancenter/mediacloud/blob/ansible_provisioning/ansible/roles/apache2-fcgi/templates/001-mediacloud.conf.j2 Apache 2 to run Media Cloud webapp, optionally with SSL
As per the book, all the rules are idempotent, meaning that Ansible tries to get the host into a specific "state" rather than blindly running a bunch of commands.
Everything runs and gets tested on Travis; test builds are now a little bit slower (because more stuff happens on the test build now), but maybe there's a way to run those builds using Docker on Travis and so speed up the test runs by using an Ubuntu image with pre-installed dependencies, don't know, haven't looked into it yet.
I took the opportunity to simplify the Media Cloud setup a little too:
-
Python's Virtualenv now gets stored in ~/.virtualenvs/mediacloud/ instead of mc-venv/ so that it can be both 1) used with virtualenvwrapper https://virtualenvwrapper.readthedocs.io/en/latest/ and 2) be cached on Travis
After some consideration, I've removed Carton and made Ansible install all the Perl dependencies using cpanm to perlbrew-system@mediacloud library under ~/.perlbrew/ because:
Carton is buggy. I've spent endless hours writing exceptions over exceptions https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_with_carton.sh to make it work. Also, quite often one needs to run carton install three times for all the dependencies to get installed.
- Carton is slow. To make the Perl script use dependencies installed with Carton, we have to use run_with_carton.sh wrapper which runs a Perl script that runs another Perl script. Why does it take 500 ms for it to set PERL5LIB is beyond me.
- Carton is hard to maintain. We've both spend yet more hours committing cpanfile.snapshot which always gets overwritten with some new stuff.
- Carton is confusing. Currently we install some Perl dependencies using APT using one version of Perl, some other dependencies "outside of Carton" https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_outside_of_carton.sh using another (Perlbrew) version of Perl, and then the majority of dependencies "with Carton" https://github.com/berkmancenter/mediacloud/blob/master/install/install_modules_with_carton.sh into yet another Perl library.
- One can't use an IDE (e.g. Komodo https://www.activestate.com/komodo-ide) with Carton.
- We don't really use Carton's main feature which is dependency version locking. I don't know about you, but whenever I add / remove a Perl dependency, I just commit cpanfile.snapshot and pray for it to work (which it usually doesn't) without giving much thought about which other dependencies got updated. I understand that Carton was introduced to have Perl dependency version locking, but I think that it has since become an outdated nuisance rather than a helpful feature.
If we absolutely positively want to have this dependency version locking (which, I think, we don't really use that much and which is likely obsolete now that we run our unit tests quite often on Travis), I'd much rather have a DarkPAN mirror on S3 http://blogs.perl.org/users/steven_haryanto/2014/01/installing-modules-from-cpan-and-your-own-darkpan.html and install hardcoded versions of everything from there rather than get back to Carton.
I've merged multiple "initialize some environment variables to use correct Perl / Python and then run something" wrapper scripts into a single run_in_env.sh. The script initializes Perlbrew environment, Virtualenv environment, and then runs whatever command was passed as its argument. This will allow us to have a single wrapper for running both Perl and Python scripts.
For example, with run_with_carton.sh one has to run a single unit test as follows:
Call
carton exec
, which will runprove
, whichrequires an additional Perl include path
./script/run_carton.sh exec prove -Ilib/ unit_test.t
With the run_in_env.sh script, one can do:
Initialize Perlbrew, add "lib/" to Perl
include path, initialize Virtualenv,
run the argument command
./script/run_in_env.sh prove unit_test.t
...or you can simply start a shell and run perl / python normally:
$ ./script/run_in_env.sh bash mediacloud$ prove unit_test.t
I hope this will save some development time for us.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/berkmancenter/mediacloud/issues/179#issuecomment-323227159, or mute the thread https://github.com/notifications/unsubscribe-auth/ABvvTx-xHUi2_tQ8b-WzXdMv_ZRxAopvks5sZNfrgaJpZM4OdgN4 .
-- Hal Roberts Fellow Berkman Klein Center for Internet & Society Harvard University
This is a pain on the front end as well, as it pertains to Javascript npm dependencies. Things were just changing too fast for us to not specific module versions. So we have our version lock file checked in and upgrade manually every month or two (we track this on a bug). Some modules are purposefully held back to older versions because newer ones would require extensive changes that don't give us much of a win. That's the world of JS right now...
Sorry for not being clear. I'm like dependency version locking, I just don't like the tool that we use for it (Carton) due to things listed in the above comment, and so I would like to replace it with some other option.
As far as I see it, the options are:
I'd go with 2 or 3.
Some more developments:
After trying out various options for installing Perl dependencies and locking their versions, I've opted for snapshotting the whole CPAN repository to S3 and installing Perl modules from it, i.e. maintaining a MiniCPAN repository of the latest versions of all the modules. This is a bit of a shotgun approach, but:
cpanm
works each and every time without having to cherry-pick which module depend on other modules.I hope this will be a better solution than Carton was, work out for us in the long term and wouldn't become a maintenance headache.
To make life with Travis easier and have a chance to try out Docker, I've build a Docker image based on ubuntu:16.04
and pre-installed with Media Cloud dependencies (system, Python and Perl ones). Travis then fetches the image, creates a test container out of it, copies the code to be tested to that container, runs the full Ansible script to update the container with the latest dependencies, and lastly executes the test suite on the container. As a result:
Main remaining task is to automatically deploy code and restart services using Ansible.
(Clicked "Comment" too early on previous comment, updated it since.)
Ansible branch merged and deployed. The summary of this Brave New World:
There's a short README describing the Ansible setup.
To set everything up on your local machine without much hassle, run ./install.sh
(which, in turn, will run the Ansible script).
Perl dependencies:
local/
with dependencies), everything gets installed into Perlbrew's perl-system@mediacloud
library under ~/.perlbrew/libs/perl-system@mediacloud/
.ansible/roles/perl-dependencies/vars/main.yml
.tools/cpan/mirror-cpan-on-s3.sh
available that will fetch all the modules from CPAN and upload them to S3 again (takes about 30 minutes and 4 GB on a good connection).FindLib
shim at the top of every script as run_in_env.sh
sets that path automatically.Python dependencies:
mc-venv/
to ~/.virtualenvs/mediacloud/
to make it compatible with virtualenvwrapper
(should you choose to use it).ansible/roles/python-dependencies/vars/main.yml
.run_with_carton.sh
, etc. has been merged into a single run_in_env.sh
script which initializes both Perl and Python environments correctly:
# Run a Perl script (with a correct '#!/usr/bin/env perl' shebang)
$ ./script/run_in_env.sh perl_script.pl
# Run a Python script (with a correct '#!/usr/bin/env python' shebang)
$ ./script/run_in_env.sh python_script.py
# Run Bash, use it to run anything you want
$ ./script/run_in_env.sh bash
mediacloud$ ./perl_script.pl
mediacloud$ ./python_script.pl
mediacloud$ exit
I'll send you a sample Ansible inventory file that you can use for deployments against our production in a private email.
Travis now pulls dockermediacloud/mediacloud-travis
with Ubuntu 16.04 and Media Cloud system / Perl / Python dependencies preinstalled, reruns the full Ansible setup script and then continues with the tests. Sometimes we might need to rebuild that test image using ansible/Dockerfile
to speed things up a little bit, but that's not going to happen too often. Travis run time is about the same.
It is (or should be) completely safe to rerun both setup.yml
and deploy.yml
playbooks multiple times against production.
There's a Slack channel #deployments
to which Ansible will send notifications about the code being deployed.
It comes without saying, but if you have a new Cron job to set up, PostgreSQL setting to toggle, Perl / Python dependency to install, Apache configuration property to adjust, etc., please set it up in an Ansible script instead of editing things on production manually.
Let me know if something doesn't work or if you have questions.
@rahulbot & Cindy need a testing server, so we'll have to maintain two servers from now on.
This is a good chance to rewrite our installation and deployment procedures to Ansible.
mecab
from APTapache stop
andapache start
into a single handleransible-lint