search

hyphacoop / organizing

Coordination and documents for our member and board meetings 📑 🌴
https://meetings.hypha.coop
11 stars 7 forks source link

Move internal infra to production env #253

Closed benhylau closed 4 years ago

benhylau commented 4 years ago

This initial comment is collaborative and open to modification by all.

Task Summary

🎟️ Re-ticketed from: #
🗣 Loomio: N/A
📅 Due date: N/A
🎯 Success criteria: Develop a process for @hyphacoop/infrastructure-wg to ensure reliability of our hosted services.

Discuss expectations and processes for:

This will probably involve moving pieces of our infrastructure into source control to start, and discussion of [ development vs. staging vs. production ] environments, adoption of monitoring and notification systems, and move us towards infra-as-code and automation tools that will let us spin up service instances rapidly.

+---------+   +---------+                    +-----------------------+
| Fact    |   | Service | (dev, stg, prd)    | Dedicated   | Proxmox |
| Finding +-->| Config  +------------------->| Server      |---------|
| Form    |   | Store   | REPO               |             |     KVM |
+---------+   +---------+  Ansible provision +-----------------------+
                           Service monitor

Details here.

To Do

ASoTNetworks commented 4 years ago

Service inventory have been updated with the latest services.

ASoTNetworks commented 4 years ago

Here are some hosting providers and products we discussed:

Dedicated

So you Start (OVH): E3-SAT-1-32 - $57.99 CAD/mo

TeraSwitch (Only US location): X5650 Dedicated Server - $50.00 USD/mo

E3-1270v6 Dedicated Server - $69.00 USD/mo

VPS:

DigitalOcean:

TeraSwitch

ASoTNetworks commented 4 years ago

I still have to learn Ansible to give a time on how long it will take.

Edited: @benhylau added resources

dcwalk commented 4 years ago

Can we include looking at companies that provide more ethical (and sustainable) hosting practices? Criteria could include:

Some examples of hosts:

There are more, some places aggregate lists: https://b2evolution.net/web-hosting/green/

benhylau commented 4 years ago

Target Architecture

Drafted this with @llunacreixent.

Admin access to all hosts should be accessed from behind a VPN, with Service VMs exposing selective ports to the public Internet. Controller VM is the control centre for all manual/CI interactions with Service VMs.

+----------------------------------------------+
| Dedicated Server                             |
|                                              |
| +---------------+ +-------------------+----+ |
| | Controller VM | | dev | Service VMs | .. | |
| |---------------| +-------------------+----+ |
| | Repositories  | +-------------------+----+ |
| | CI builder    | | stg | Service VMs | .. | |
| +---------------+ +-------------------+----+ |
| +---------------+ +-------------------+----+ |
| | Monitoring VM | | prd | Service VMs | .. | |
| +---------------+ +-------------------+----+ |
+----------------------------------------------+

Repositories:

Environments & Workflow:

  1. <service>-<uuid>.dev.hypha.systems is deployed anywhere from behind VPN
  2. <service>-<org>.stg.hypha.systems is deployed by CI manually/on-merge from CONF/staging branch
  3. Manually verify staging features on stg environment before merging to master
  4. <service>.<org>.hypha.systems (or custom domain), aka. prd, is deployed by CI manually/on-merge from CONF/master branch

MVP Architecture

These are the differences from above target architecture:

Open Questions

  1. What exactly goes into PLAY vs. CONF
  2. How to "add Inventory" (scale beyond one dedicated server)
  3. Backup strategy (MVP and beyond)
  4. How to ensure others can make use of PLAY despite not having CONF (a sample?)
benhylau commented 4 years ago

@ASoTNetworks some scenarios to consider, would like a step-by-step from you:

  1. Provision 3 new Jitsi instances
  2. Update firewall rules on 2 of 3 Jitsi instances
  3. Dedicated server self-destruct and we need to re-instantiate the whole stack and with backup
benhylau commented 4 years ago

Adding to list of providers, a Montreal based coop we met at meet.coop's call https://www.koumbit.org/en/services/vps

benhylau commented 4 years ago

Internal workflow and their current repos

Proposal by @benhylau

Updated after reviews by @hyphacoop/infrastructure-wg

  1. Consistent namespace for infra repos that hold generic code without configurations (e.g. hyphacoop/infra-shortlinks, hyphacoop/infra-vision)
  2. Public configuration repo that has a dependency on above, and hold any custom config and deployment-specific files (e.g. BBB room config, matrix room names, shortlinks.csv)
  3. Private secrets repo that holds deployment keys and passwords (e.g. API keys, credentials)

Controller VM above would pull all the repos and deploy into prod env.

CI deploys into stg env. We can spin up VMs in dev env as needed.

benhylau commented 4 years ago

Timeline

benhylau commented 4 years ago

We need something like this:

Screen Shot 2020-05-28 at 1 31 56 AM

This is with soyoustart.

A similar system with a Canadian green host will be the Dedi Pro with whc.ca but need to add an additional $55/month for 16 IPs and 100 GB of backup space, which brings the monthly cost to $218.94. Comes with CentOS.

benhylau commented 4 years ago

Since we have already received the quote from Koumbit, I'd like to ask we make a final decision on server vendor on Wednesday's call.

The above soyoustart server and discount is no longer available, here are some comparable ones:

Note that all these OVH and soyoustart servers are in the Beauharnois datacentre and hydropowered (with 2 other sources as power redundancy):

ASoTNetworks commented 4 years ago

After looking at the above products I like the OVH rise-1 option more as it comes with newer hardware, IPMI access, and 500G of backup space that can be mounted over Samba and NFS.

ASoTNetworks commented 4 years ago

I have updated our internal server comparison spreadsheet highlighted with colors of the top 3 choices we looked at.

Server comparisons sheet

benhylau commented 4 years ago

@ASoTNetworks can you also fix the infra ranks? They seem to not be consistent what 1-5 means. Also some of the costs seem wrong with setup cost and stuff.

ASoTNetworks commented 4 years ago

Fixed it and only rated the ones that we can use.

benhylau commented 4 years ago

This is a huge task. I have made a milestone to track this so we can close this thread off.