Dornkirk Migration by 30 Sept 2021

adiroiban commented 3 years ago

This is a todo list of things that needs to be done to migrate Dornkirk

Changes for braid are at https://github.com/twisted-infra/braid/pull/279

To access the new VM

ssh azureuser@twisted-vm.centralus.cloudapp.azure.com

Backup at https://twistedbackup.blob.core.windows.net/dornkirk

General tasks

[X] Azure it is - Find a reliable cloud provider to use it in the near future - https://twisted-vm.centralus.cloudapp.azure.com
[x] See how we can migrate the Twisted Azure subscription with 150$ monthly credit to some Twisted org. Migrated.
[X] Rackspace Credit - Twisted Leadershit Team is not active so we can't do much. See what is the status with Rackspace monthly credit and if we can rely on it. Maybe review this once the new leadership team is active.
[X] Plan B - Migrate the whole dornkirk to an Azure VM
[x] Take a backup/snapshot of Dornkirk before closing it
[x] Check with tummy to see how we can get old backups... I asked, I got no answer but we should be fine.

Backup

[x] root home
[x] /home (users)
[x] /etc
[x] amptrac
[x] bb-master
[x] codespeed
[x] diffresource
[x] highscore
[x] kenaan
[x] t-names
[x] trac
[x] t-web
[ ] root volume
[x] /var (postgres and maybe other services)
[x] Azure Blob / Files - Find a place to store the backup and move the backups there

t-names DNS tasks

[x] Migrate the whole twistedmatrix.com DNS to Gandi.net
[x] Migrate the other zones to Azure and start DNS server (open firewall) - crookedsapling.us divmod.com divmod.org divunal.com ynchrono.us
[x] Update ns1.twistedmatrix.com - Done by Glyph in Gandi
[x] Update the zones configuration

Mailing list

[x] Migrate the mailing list operation to Python infrastructure
[x] Send the ML config and archive files to Mark
[x] Announce the movement to the old ML
[ ] Migrate the mail-archive to Azure VM - https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[ ] Or implement a redirect to Python hosted ML - https://mail.python.org/mailman3/lists/twisted.python.org/
[ ] Update our documentation and pages to point to the new ML page
[ ] Arrange for Twisted mailing list archive redirection ... or do a simple legacy static HTML archive.
[ ] Migrate the Twisted Software Foundation list to mailgun
[ ] Just backup/archive cube-administrators

bb-master Buildbot

[ ] Option A - Migrate buildbot jobs to GitHub Actions - https://buildbot.twistedmatrix.com/buildslaves
[ ] Option B - Migrate buildmaster on the same VM as Trac Plan B
[ ] Migrate CodeSpeed - https://speed.twistedmatrix.com/

Trac

[x] Option A - Migrate trac to a VM in the cloud
[ ] Option B - Migrate trac tickets to GitHub Issues
[ ] Check that weekly ticket activity summary still works (optional)
[x] Migrate all the wiki pages to something like twisted/twisted.wiki repo https://github.com/twisted/twisted/wiki/TwistedCommunity
[ ] Convert TracWiki wiki c ontent to MediaWiki or RST or Markdown
[ ] Move important information from the wiki, into Sphinx narrative-docs

t-web

[x] Create a simple static page on https://github.com/twisted/twisted.github.io as the start page
[x] Migrate old API docs (not available on RTD) https://github.com/twisted/documents - https://twisted.org/documents/21.2.0/
[x] Migrate twisted-web code

amptrac

[x] Plan B - migrate to Azure VM

codespeed

[ ] Plan B - migrate to Azure VM

diffresource

Will be decommissioned

[X] Archie to Azure Blob

highscore

[ ] Plan B - migrate to Azure VM

kenaan - IRC / Gitter / Matrix

[ ] Plan B - migrate to Azure VM
[ ] See what to do with IRC bot ... maybe just kill it and do it via Gitter IRC bridge

adiroiban commented 3 years ago

A gitter / matrix bride was setup between Libera #twisted channel and Gitter https://gitter.im/twisted/twisted

so we kind of have logs for the IRC channel via this bridge

adiroiban commented 3 years ago

I think that by this time, we need to trigger plan B.

Try to migrate as much as possible from the current machine (which I think it's a 2xCPU bare metal beast) to a new cloud VM with virtual CPUs :)

adiroiban commented 3 years ago

So... Plan B is Azure as it should be the most probable VM hosting from which we can get some hosting sponsorhip.

Rackspace is out of question as twisted leadership team is active ...so we can't do much for now.

DigitalOcean / Vultr are nice, but setting up billing is complicated...so I think that is OK to go with Azure.

The long term plan is to remove the VM and migrate everything to GitHub / Read The Docs

glyph commented 3 years ago

Thanks @adiroiban ! We can restore other services as time allows. I pinged on IRC but since you may not be there, I'm just wondering if my SSH key is supposed to be on the new host? I tried to ssh in but I couldn't do so as either glyph@ or root@.

adiroiban commented 3 years ago

I tried to ssh in but I couldn't do so as either glyph@ or root@.

ssh azureuser@twisted-vm.centralus.cloudapp.azure.com

It's in the description 😃

adiroiban commented 2 years ago

@glyph have you tried ssh azureuser@twisted-vm.centralus.cloudapp.azure.com ?

I have copied the 5 SSH keys that I found on dornkirk

glyph commented 2 years ago

It doesn't work. I could ssh to dornkirk just fine, so perhaps they were corrupted by the addition of some whitespace.

You can grab current SSH keys for me here: https://github.com/glyph.keys

adiroiban commented 2 years ago

I have updated the SSH keys from GitHub.

There were no whitespaces in the previous keys.

Try now.

glyph commented 2 years ago

Works now.

adiroiban commented 2 years ago

We are still doing the dornkirk migration as not all the services were migrate.

With the current resources, I think that our only hope to success is go serverless and use as many SaaS service as possible for the infrastructure.

This mean that the whole twisted-infra org should be close/deprecated as we will no longer need to care about the infrastructure.

We will just used other people infrastructure.

We have already started this process, with the git source code hosting with GitHub, and now with the mailinglist.

We are also using Read the Docs to host our documentation.

I guess that moving from Trac Ticket to GitHub Issues is the next big move.

And then look at using GitHub Pages or Netlify to host all our web presentation pages/blog.

So, with that in mind, I will try to create all the "infra" related issues inside repos hosted by https://github.com/twisted

We can still keep this org and repos for historical reasons.

glyph commented 2 years ago

With the current resources, I think that our only hope to success is go serverless and use as many SaaS service as possible for the infrastructure.

Don't we still need a place to keep the SaaS configuration? If we just move from manual configuration on a single host to manual configuration with random bespoke logins on various services it doesn't seem like much of an improvement in terms of administrative overhead.

adiroiban commented 2 years ago

The SaaS configuration should be kept in the repo source code.

This is what we are already doing for twisted/twisted

The documentation configuration is stored at https://github.com/twisted/twisted/blob/trunk/.readthedocs.yml

The buildbot replacement is stored at https://github.com/twisted/twisted/tree/trunk/.github/workflows and https://github.com/twisted/twisted/tree/trunk/azure-pipelines

And for Github pages or Netlify you have similar stories. The configuration and content is stored in the same repo.

But we should try to use as fewer logins as possible... that is, try to build everything around GitHub infra.

I know that in theory it sound like a terrible idea, but with the limited resources, I think its our only hope to move forward.

And we should have a little configuration as possible :)

And regarding the Azure VM. I have only did the migration as plan B. Since the migration, I have only checked the VM once to enable backup snapshots and since they I have not checked it again. I have not checked to see that the backups really work and I don't know if anyone else had checked this.

adiroiban commented 2 years ago

I think that this can be closed.

We plan to turn off the VM by the end of this year... without any plans to add more services to the VM.

Codespeed can be implemented in a server-less / container environment.

adiroiban commented 2 years ago

some legacy mailing list archive were migrated here https://twisted.org/pipermail/ and hosted via GitHub Pages

the full archive is stored on Azure Blob

twisted-infra / braid