Go single region

[ycombinator]

via @kenperkins: The benefit of multi-region doesn't outweigh the added complexity for our product. Consider:

1. Losing an entire region is a catastrophic level failure.
2. Our current strategy uses round-robin-dns, which means in the event of the afore-mentioned region outage, half of our customers are going to get connection errors until the DNS cuts over.
3. We can provision an entire new deployment in a second region in minutes
4. We don't have sufficient data-gravity on a per-region basis to impact restoring to full functionality after provisioning in new region

Given these statements, I believe we should do-away with the multi-region deployment, and switch to a single region.

[rdodev]

With many other non-critical businesses doing the same (i.e. implement in one region, provision other regions as disaster recovery), it seems this is a good way to handle disaster recovery for our use case.

:+1:

[ycombinator]

1. We don't have sufficient data-gravity on a per-region basis to impact restoring to full functionality after provisioning in new region.

Any time we add a datastore (cloud files containers, elastic search cluster, etc.) we must have a strategy that ensures:

• that this datastore is populated when we failover to a new region. This population does not have to be instantaneous.
• that we can migrate data from the datastore in the old region to the one in the new region once the old region is available again.

[ycombinator]

Execution note: We cannot simply kill one of our existing regions to resolve this issue. Currently we are storing sponsorship requests in containers in IAD (hardcoded). We would want to make the following changes first:

• change the appropriate Ansible playbooks to ensure that sponsorship containers get created in the same region as the Ansible deployment, and
• change our sponsorship form processing code to write to the containers in the same region as where the code is running.

Alternatively we could just use ElasticSearch as the backend for sponsorship requests, eliminating two types of data stores (cloud files and elasticsearch).

[rdodev]

@ycombinator you want this to be a backlog item for next sprint?

[ycombinator]

@rdodev Yes, I think that makes sense. We would be able to make this change before introducing more changes (e.g. ElasticSearch).

[rdodev]

@ycombinator ok, I'll add the items delineated above.