mekarpeles
commented
5 years ago This task needs a checklist:
- [x] Prove provisioning a
genericminimal xenial VM (e.g. of theol-memflavor) and add it to theolcluster (e.g. asol-mem4) - [x] Codify the
ol-memapproach using Ansible @abezella -- following https://github.com/internetarchive/openlibrary/issues/680#issuecomment-481929471 - [x] Prove provisioning of an
openlibrary-specificxenial VM (e.g.ol-web3) using Docker and add it to theolcluster (e.g. asol-web1). This requires preserving the existing/opt/openlibrarydirectory as a detachable volume and preserving to new instance. - [x] Codify the
openlibrary-specificapproach (using Ansible + Docker) such that newol-webxenial nodes can be added automatically into theolpool.
tfmorris
cclauss
jimman2003
Related to #703 (see aspirational #680)
genericminimal xenail VM (e.g. of theol-memflavor) and add it to theolcluster (e.g. asol-mem4)ol-memapproach using Ansible @abezella -- following https://github.com/internetarchive/openlibrary/issues/680#issuecomment-481929471openlibrary-specificxenial VM (e.g.ol-web3) using Docker and add it to theolcluster (e.g. asol-web1). This requires preserving the existing/opt/openlibrarydirectory as a detachable volume and preserving to new instance.openlibrary-specificapproach (using Ansible + Docker) such that newol-webxenial nodes can be added automatically into theolpool.View Architecture & Provisioning docs on the Wiki
Remaining Trusty Machines
Requirements
The Trusty 14.04 release of the Ubuntu operating system will reach end of life for LTS (long-term support) at the end of 2019. After this time, our VMs may no longer receive necessary security updates. Therefore, before 2020, we are required to re-provision all our ~11 production Open Library VMs to run Xenail.
Current Production Architecture
Today, our production service architecture consists of ~11 VMs:
(see: https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture)
Current Provisioning Setup
Our current production setup process (as of 2019) for provisioning these 11 VMs is ostensibly manual and relies on a lot of manually
scping directories around, as well as a separate repository calledolsystemwhich contains the production configs, cron jobs, and infrastructure required to run the official openlibrary.org service.Each of our 11 VMs are more-or-less provisioned identically:
/optdirectory containing all the "business"/optthere is anopenlibrary/and apetabox/directory. It's very likely/opt/petaboxis not required by all VMs, though it's not currently well understood which services may rely on it (e.g. theol-homeVM makes heavy use ofolsystemwhich may reference petabox)/opt/openlibrarycontains all the business logic for the Open Library project:Minimum Proposal
At minimum, re-provisioning a VM requires:
scp'ing over the legacy VM's/optdirectory (preferably as an external mountable/1volume which can be moved in the future)olsystemso that its files within/opt/openlibrary/olsystem/etcsymlink to the right locations within/etcTo copy
/optover from another server you'll have to: onol-mem2:sudo tar cpSlf /var/tmp/ol.tar --same-owner -C /opt openlibraryscp /var/tmp/ol.tar ol-mem4:/var/tmp/ol.taronol-mem4:tar xpBsf /var/tmp/ol.tar --same-owner -C /opt(edited) (due to keys and needing to be root to get all of it i don't there's an easy way to just scp or rsync)Ideal Proposal
An aspirational goal of this epic is to migrate Open Library VM provisioning to use a standard Ansible playbook (and possibly docker containers, a la our development environment) to support this re-provisioning.
Part of this effort includes decreasing production's dependence on the
olsystemrepository a la #680. Both developer and production systems should use have similar docker recipes and differ according to ansible playbooks.Plan
The plan is to start with
ol-mem0,ol-mem1, andol-mem2as they don't really require any infrastructure other than: 1) setup 3 new memcached serversol-mem3,ol-mem4,ol-mem42) provision VMs with default ansible playbook: setup firewall rules + install docker 3) use VM-specific ansible playbook to install setup docker w/ memcached (with upstart) 4) update /opt/openlibrary/olsystem/etc/openlibrary.yml and infobase.yml configs to reference correct new memcached servers 5) /etc (e.g. memcached) to symlink to the correct system configs in/opt/openlibrary/olsystem/etc/6) update /opt/openlibrary/olsystem/fabfile.py supervisord to update how memcached servers should be restarted (and to not deploy to ol-mem* during deploy) 7) remove old memcached servers from the pool (one at a time)