Standard Change - Purchase progcode Cloud Hosting Service

UPDATE - MARCH, 15, 2023

Updated as a Standard Change to capture additional cloud hosting provider needs

Hosting Service	Owner	Date	Comments
Digital Ocean	@stephenscapelliti	March 8, 2019	Previously implemented per below discussion
Azure	@jpb5013	March 15, 2023	New cloud service as part of community operations for Project.Gallery Project

Description

~This is a proposal to purchase a 1GB “Standard Droplet” account from Digital Ocean for access by ProgCode Operations Staff and other community members in fulfillment of the ProgCode mission, with the option of continuing the account, increasing capacity at a later date, or cancelling the account.~

This is a proposal to establish a budget for Digital Ocean in the amount of up to $15/month for one 1gb/1core server ("Standard Droplet") and one 2gb/2core server or any similar combination within that budget, and to allow the Directors discretion to adjust that amount downward - with community input - if experience indicates that usage requires a lower expenditure, with the option of continuing the account or cancelling the account.. Any increase above $15/month would require an amendment to this issue and a community vote to implement. - This issue was amended on 2019.04.06 pursuant to the Community Operations meeting on 2019.04.01.

Problem

ProgCode has relied on Heroku for hosting of the ProgCode website and other projects (such as Progbot, Toolkit scrapers, PhoneYourRep), due to the available features. Heroku is affordable and cost-effective for our particular uses. Digital Ocean and other hosts offer a more-cost effective hosting option for projects which do not require the same features offered by Heroku. Recent discussions have included expansion and revision of the ProgCode website, the establishment of a chat resource (such as Discourse), and a searchable Slack archive backup.

Benefit

A Digital Ocean account would enable ProgCode to implement some of the above-mentioned projects to benefit the community at a lower cost than if we increased usage of Heroku for those same projects. A minimum plan of 1GB will cost $5/month or $60/year. It can be increased, if necessary, to 2GB, which will cost $10/month or $120/year. A discount is offered to non-profit organizations, which might reduce the cost further, if ProgCode qualifies.

Expenditure Analysis

This proposed solution would require a minor annual expenditure (between $60/year for 1GB and ~$120/year for 2GB~ $180/year for one 1gb/1core server and one 2gb/2core server or similar combination - amended 2019.04.06).

The requirements of Issues #198 and #236 are applicable, as follows:

Proposals seeking consent for implementation of budget requests and/or change process should not contradict ProgCode core objective and should pass this four-pronged test when considered by operations:

Is this proposal non-partisan in nature? Yes
Is this proposal aligned with ProgCode's mission of removing the influence of big money in politics? Yes
Does this proposal seek to empower the grassroots by supporting the creation of open source tech? Yes
Is this proposal sponsored or funded by an outside organization? If yes, the source of the funding must be disclosed. No

Plan

[x] Community discussion of all elements of this proposal at the 03.11.2019 Community Operations Meeting.
[x] Vote for consent to ~~implement~~ proceed to investigate the capacity needs for the standard change to purchase a month-to-month 1GB Standard Droplet Digital Ocean account for access by all ProgCode Staff and community members assisting the staff, for the benefit of and use by the ProgCode community.
[x] Vote for consent to proceed for the standard change to purchase the capacity needed from Digital Ocean at a later date.
[x] Notify the community of the vote taken by posting a #team-announcement and sharing the video URL in #operations immediately following the 03.11.2019 Community Operations Meeting to allow additional votes & feedback during the standard 72-hour waiting period before ratifying the vote.
[x] Vote for consent to implement the standard change to purchase the capacity needed from Digital Ocean at a later date.
[x] If vote for consent to implement passes, purchase a month-to-month Standard Droplet Digital Ocean account in for the ProgCode Administration account in whatever capacity is approved in a community operations meeting.
[x] Grant login access to the Digital Ocean account to all current and future ProgCode Staff members, and to others assisting staff, on an as-needed basis.
[ ] Staff will investigate non-profit and other discounts which may be available to ProgCode and make application as needed to qualify.
[ ] Operations Staff to create a standard process for use, maintenance, and security of the ProgCode Digital Ocean account.
[ ] Keep this issue open for possible amendment a later date, for continuing the account, increasing capacity at a later date, or cancelling the account.

Decision Making

Consent to implement a standard change per the Change Process

Optional Information

Reference link(s)

Community Discussion History:

02.26.2017 - Issue 81 "Funding Framework"
05.15.2017 - Issue 198 "Four Pronged Test for Mission Alignment"
06.18.2018 - Issue 236 "Standard Change - Budget Expense Requests"
02.25.2019 - Issue #253 "Upgrading our Tech Stack and available hosting resources"
03.07.2019 - Discussion of Digital Ocean at Progbot meeting beginning at the 27:20 mark.

So, I setup Discourse on a personal VM instance of mine. The VM was setup to match the specs of the $5 DO droplet. 1 core and 1gb of ram.

While discourse does in-fact run on it, the recommended production setup using docker leaves very little resources left to do anything else on the box. I also a lot of doubts that this setup would be enough to run discourse on a production load. I'm also positive that if we run discourse and any nodejs bot dedicated to logging our slack, we will without a doubt run into resource contention issues which will be highly noticeable. Here are metrics that I gathered while bringing up the environment; https://i.imgur.com/l9bo6xJ.png

The CPU usage during the deployment of the docker container is normal, really. Anytime the docker container needs to be rebuilt (due to a config change or update) the CPU will spike again. As you can see from the graph, the CPU spiked past 100% utilization quite a few times. When this happens it has the potential to negatively affect other things running on the box (such as an archiver for slack). In addition to that, my fear with CPU usage would be cpuwait causing the updates to malfunction or just take an unnecessarily long time.

Here is a metric readout that displays user activity. This load test was a very simple test. It simulated 25 users loading the discourse home page once. They didn't perform any dynamic actions which would result in bith higher ram AND cpu usage(ie a user logging in, registering, or replying to a post, checking their PM's.) so ram usage in a real use scenario would be higher. The part highlighted in green represents the time I started the load test.

https://i.imgur.com/alyKmAN.png

Each connection to discourse uses between 0.5mb to 1mb of ram with no dynamic actions, so in a real use scenario, 25-50 connections onto the forums could memory lock the box in this case, causing the machine to go into swap memory (which most instances don't have). Each connection has a "stay-alive" period of several minutes, so "busy" spikes will show this limitation even more. We could create a swap file on the hard drive to combat this, but that presents a host of issues on it's own and isn't a supported configuration by discourse. Swap is also MUCH slower than in ram paging. In-fact it's been frowned upon and in many cases discourse support-team has rejected support requests when that configuration is used. Memory limits of rails can be limited inside the docker container however this will degrade performance greatly for users and cause other issues as well.

The following graph shows the response time of the client vs's the number of connections loaded. My statement above is reflected in this graph as you can see response times become slightly less consistent as the number of test nodes go up.

https://i.imgur.com/uwXT7QP.png

I'm going to suggest that in order to give our users a good user experience and allow us to use this droplet for more than just a forum that we go with the $15 a month 2gb 2 core droplet. This will give us enough room to allow discourse to be used within a production environment without hindering other services that may be running on that box. Also note, the specs I'm recommending are what I originally recommended on March 7th. https://progcode.slack.com/archives/C3DRQ90R0/p1551999490030100

It might make sense to use Heroku to try out the Slack archive and Discourse, so that we can determine our needs, and then amend this proposal before a vote.

Discourse and the Slack Archiver run via Docker, and thus cannot be run in a Heroku Dyno. Losing the docker layer is not recommended as we loose support options for discourse (they'll tell us to try it in the production docker deployment. Non-docker is meant for development). The slack Archiver doesn't have an easy way to run outside of docker either, since the applications that make up the entire suite are build to rely on docker as an underlying way to configure it's distributed model.

It's probably wise that instead of discussing authorization for a specific droplet size, we discuss an account budget instead, with the intention to use the least amount of that budget per month, and let the remainder rollover into the next months. This would allow us to use the minimal amount of resources needed to run any specific project/app, but also allow us to quickly scale up if we see a spike in resource usage.

I've ran into many situations where it was discovered that the minimal droplet size wasn't enough capacity for a number of reasons (sudden burst of traffic, underestimates of requirements, addition of needed apps).

I think a $15 to $20 a month budget would be a good starting point. $20 a month would allow us to run One 1gb/1core server and One 2gb/2core server, or scale either of those options in the event that we hit a spike in traffic. As the months continue on, as long as we haven't had a need to scale, the rollover of the remaining budget would increase our scaling options.

To give you an idea of the type of resource usage we could see; I run feelthebern.org's production site and staging site on two separate 1 core, 1 gb instance. The instance is constantly at 65% used ram for both instances, with prod spiking to over 90% on high loads. Prods CPU is generally idle at 33% with 7 users on the site, and can spike too over 70% on high loads (over 50 users). Both servers are running nginx, php7.3-fpm and mariadb in an optimized configuration as well as Bro for Intrusion Detection. We also have another server that's 2cores 2gb of ram that's solely for running our gitlab instance. This server sits generally around 85% ram usage, and 2-5% CPU.

We pay $25 a month for that setup, and will have to scale out when the busy season hits.

I think setting approval for a specific droplet size is a great way to back us into a corner with a non-working app/site once we hit capacity limits. It's best to plan ahead given that we also will be facing a busy season soon.

Pursuant to the discussion during the Community Operations meeting on 2019.04.01, as well as Alex's research referenced above, this issue will be amended to seek consent to implement a budget for Digital Ocean in the amount of up to $15/month and to allow the Directors discretion to adjust that amount downward if experience indicates that usage requires a lower expenditure. Any increase above $15/month would require an amendment to this issue and a community vote to implement.

ProgressiveCoders / functions