Ensure all our infrastructure is redeployable

hellais commented 5 years ago

As part of this activity we are going to ensure that it's possible to redeploy all of our infrastructure and also we will be upgrading the base OS to debian 10 (see: https://github.com/ooni/sysadmin/issues/287).

The checklist of hosts that need to be upgraded or checked that it's possible to redeploy is:

[mia]

[x] c.web-connectivity.th.ooni.io
[x] miatorth.ooni.nu
[x] pipeline.test.ooni.nu
[x] miaapi.ooni.nu
[x] miacollector.ooni.nu

[hkg]

[x] irc-bouncer.service.ooni.io
[x] ooni-zoo.infra.ooni.io
[x] hkgwebconnectivitya.ooni.nu
[x] hkgjump.ooni.nu
[x] hkgsuperset.ooni.io
[x] hkgtorth.ooni.nu
[x] labs.ooni.io
[x] jupyter.ooni.io
[x] hkgbouncer.ooni.nu
[x] datacollector.infra.ooni.io
[x] hkgmetadb.infra.ooni.io
[x] hkgcollectora.ooni.nu

[ams]

[x] amstorth.ooni.nu
[x] get.ooni.io
[x] a.http.th.ooni.io
[x] b.web-connectivity.th.ooni.io
[x] c.echo.th.ooni.io
[x] slack.openobservatory.org
[x] demo.probe.ooni.io
[x] msg.ooni.io
[x] notify.proteus.test.ooni.io
[x] events.proteus.test.ooni.io
[x] registry.proteus.test.ooni.io
[x] db-1.proteus.test.ooni.io
[x] proteus.test.ooni.io
[x] amsmatomo.ooni.nu
[x] notify.proteus.ooni.io
[x] events.proteus.ooni.io
[x] registry.proteus.ooni.io
[x] run.ooni.io
[x] proteus.ooni.io
[x] api.ooni.io
[x] amsexplorer.ooni.nu
[x] c.collector.ooni.io
[x] db-1.proteus.ooni.io
[x] amsmetadb.ooni.nu
[x] ssdams.infra.ooni.io
[x] prometheus.infra.ooni.io

The code for these services is: https://github.com/ooni/orchestra.

This is also a great opportunity to cleanup hosts that are no longer used, migrate things across datacenters, fixup domain names (see: https://github.com/ooni/sysadmin/issues/346) and also consolidate the probe related backends on one host per location.

hellais commented 5 years ago

I am not going to redeploy the tor test helpers as we will get assigned a different IP and the goal of those test helpers is to have a machine we control that has been for a long enough time in the tor consensus for doing testing of tor related blocking.

The hosts I speak of are: hkgtorth.ooni.nu, amstorth.ooni.nu, miatorth.ooni.nu.

hellais commented 5 years ago

I moved msg.ooni.io to be hosted under our primary website and setup some redirects so that old pages will still work.

This means one less machine to run and maintain!

See: https://github.com/TheTorProject/ooni-web/pull/294

hellais commented 5 years ago

I have retired the labs.ooni.io host and similarly to msg.ooni.io setup redirects so that visits will land on a page explaining how they can still get the code for labs.ooni.io and how to contact us.

See: https://github.com/TheTorProject/ooni-web/pull/300

hellais commented 5 years ago

There are still these hosts that should be at some point redeployed:

db-1.proteus.ooni.io
amsmetadb.ooni.nu
ssdams.infra.ooni.io
prometheus.infra.ooni.io
hkgmetadb.infra.ooni.io
datacollector.infra.ooni.io

But I am thinking it’s probably wise to do those at some later point with more calm.

For the database hosts it’s maybe an opportunity to look into upgrading to a newer version of postgres which is tricky since postgres 9 does not support logical replication and is basically impossible to do with zero downtime. See: https://github.com/ooni/sysadmin/issues/383

For the monitoring, I don’t think we want to touch it as it works and it helps us understand if we broke something in this mass redeploy/. See: https://github.com/ooni/sysadmin/issues/384

And the pipeline running on datacollector, well, I don’t fully understand how it works. I have made changes to the role so that it applies cleanly to the running host, but I don’t think it’s wise to do that in a rush either. See: https://github.com/ooni/sysadmin/issues/385

hellais commented 5 years ago

Closing as this is done.

ooni / sysadmin

Ensure all our infrastructure is redeployable #356