Open mekarpeles opened 5 years ago
This task needs a checklist:
generic
minimal xenial VM (e.g. of the ol-mem
flavor) and add it to the ol
cluster (e.g. as ol-mem4
)ol-mem
approach using Ansible @abezella -- following https://github.com/internetarchive/openlibrary/issues/680#issuecomment-481929471openlibrary-specific
xenial VM (e.g. ol-web3
) using Docker and add it to the ol
cluster (e.g. as ol-web1
). This requires preserving the existing /opt/openlibrary
directory as a detachable volume and preserving to new instance.openlibrary-specific
approach (using Ansible + Docker) such that new ol-web
xenial nodes can be added automatically into the ol
pool.Since Ubuntu 18.04 Bionic Beaver has been out for over a year, would it make more sense to skip Xenial?
Our Xenial Docker-based development environment is producing dire warnings Node.js and the bundled version of pip doesn't work. I think we should upgrade our dev environment to Bionic ASAP in preparation for a production move to Bionic.
% cat ./ubuntu_versions.sh
#!/bin/bash
# Which Ubuntu release are we running on? Do not fail if /etc/os-release does not exist.
# cat /etc/os-release | grep VERSION= || true # VERSION="20.04.1 LTS (Focal Fossa)"
SERVERS="ol-backup0 ol-covers0 ol-db1 ol-db2 ol-dev0 ol-dev1 ol-home ol-home0 ol-mem0 ol-mem1 ol-mem2 ol-solr0 ol-solr1 ol-web1 ol-web2 ol-www0"
parallel --quote ssh {} "hostname --short ; cat /etc/os-release | grep VERSION= ; docker --version ; docker compose version || true" ::: $SERVERS
Some of the PPAs might have deleted the xenial debs/packages so the CI is failng.. So we should bump this up?
https://github.com/internetarchive/openlibrary/issues/2036#issuecomment-859350115
@dhruvmanila, @cclauss, or @BharatKalluri -- is this one you may have a few minutes to quickly investigate? If it seems like it may be a pain, @cdrini and I can prioritize for next week. @cdrini is currently PTO and I'm getting my 2nd covid shot tomorrow and will likely be out of commission for at least some of the weekend :grimacing:
It would be important to look as upgrading both:
On ol-mem0...
sudo apt-get install memcached && memcached --version
# --> memcached 1.5.22https://github.com/internetarchive/olsystem/search?q=memcache
ol-mem
https://internetarchive.slack.com/archives/G019YBYM35M/p1602178331011000
https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture
I think we're close:
ol-db[1,2], ol-backup, ol-www0->1, and presumably ol-home disappears if we can ~re~move stats-solr
(?)
@cclauss ?? :|
Using the script at https://github.com/internetarchive/openlibrary/issues/7676#issuecomment-1480274108
ol-home0% ./ubuntu_versions.sh
Related to #703 (see aspirational #680)
generic
minimal xenail VM (e.g. of theol-mem
flavor) and add it to theol
cluster (e.g. asol-mem4
)ol-mem
approach using Ansible @abezella -- following https://github.com/internetarchive/openlibrary/issues/680#issuecomment-481929471openlibrary-specific
xenial VM (e.g.ol-web3
) using Docker and add it to theol
cluster (e.g. asol-web1
). This requires preserving the existing/opt/openlibrary
directory as a detachable volume and preserving to new instance.openlibrary-specific
approach (using Ansible + Docker) such that newol-web
xenial nodes can be added automatically into theol
pool.View Architecture & Provisioning docs on the Wiki
Remaining Trusty Machines
Requirements
The Trusty 14.04 release of the Ubuntu operating system will reach end of life for LTS (long-term support) at the end of 2019. After this time, our VMs may no longer receive necessary security updates. Therefore, before 2020, we are required to re-provision all our ~11 production Open Library VMs to run Xenail.
Current Production Architecture
Today, our production service architecture consists of ~11 VMs:
(see: https://github.com/internetarchive/openlibrary/wiki/Production-Service-Architecture)
Current Provisioning Setup
Our current production setup process (as of 2019) for provisioning these 11 VMs is ostensibly manual and relies on a lot of manually
scp
ing directories around, as well as a separate repository calledolsystem
which contains the production configs, cron jobs, and infrastructure required to run the official openlibrary.org service.Each of our 11 VMs are more-or-less provisioned identically:
/opt
directory containing all the "business"/opt
there is anopenlibrary/
and apetabox/
directory. It's very likely/opt/petabox
is not required by all VMs, though it's not currently well understood which services may rely on it (e.g. theol-home
VM makes heavy use ofolsystem
which may reference petabox)/opt/openlibrary
contains all the business logic for the Open Library project:Minimum Proposal
At minimum, re-provisioning a VM requires:
scp
'ing over the legacy VM's/opt
directory (preferably as an external mountable/1
volume which can be moved in the future)olsystem
so that its files within/opt/openlibrary/olsystem/etc
symlink to the right locations within/etc
To copy
/opt
over from another server you'll have to: onol-mem2
:sudo tar cpSlf /var/tmp/ol.tar --same-owner -C /opt openlibrary
scp /var/tmp/ol.tar ol-mem4:/var/tmp/ol.tar
onol-mem4
:tar xpBsf /var/tmp/ol.tar --same-owner -C /opt
(edited) (due to keys and needing to be root to get all of it i don't there's an easy way to just scp or rsync)Ideal Proposal
An aspirational goal of this epic is to migrate Open Library VM provisioning to use a standard Ansible playbook (and possibly docker containers, a la our development environment) to support this re-provisioning.
Part of this effort includes decreasing production's dependence on the
olsystem
repository a la #680. Both developer and production systems should use have similar docker recipes and differ according to ansible playbooks.Plan
The plan is to start with
ol-mem0
,ol-mem1
, andol-mem2
as they don't really require any infrastructure other than: 1) setup 3 new memcached serversol-mem3
,ol-mem4
,ol-mem4
2) provision VMs with default ansible playbook: setup firewall rules + install docker 3) use VM-specific ansible playbook to install setup docker w/ memcached (with upstart) 4) update /opt/openlibrary/olsystem/etc/openlibrary.yml and infobase.yml configs to reference correct new memcached servers 5) /etc (e.g. memcached) to symlink to the correct system configs in/opt/openlibrary/olsystem/etc/
6) update /opt/openlibrary/olsystem/fabfile.py supervisord to update how memcached servers should be restarted (and to not deploy to ol-mem* during deploy) 7) remove old memcached servers from the pool (one at a time)