Singapore Slump

[CowMuon]

Description

As we know, app performance in Asia is significantly degraded, is the source of considerable complaint, and is simply unacceptable.

Engineering Requirements

Stated engineering assumption that is is the round trip database queries back to database North America that is main cause of performance degradation.

Wiki

https://github.com/hicommonwealth/commonwealth/wiki/geo-replication

Acceptance Criteria

• [ ] Confirm assumptions around database query round trips
• [ ] Document proposed solution (inter-regional replication)
• [ ] Implement working solution and benchmark measured improvement.

[NakulManchanda]

@CowMuon Can we post how are we measuring and what are we measuring? And what are the current numbers - which we are improving upon?

To confirm database round trip query:

• we can focus on just response time from routes only
• other assets which are not served from database should be ignored

Ideally, same user need to measure both numbers

• may be we can use Malik he is in south-east asia

[NakulManchanda]

And to experiment with numbers:

• we can deploy new app in Singapore region - if Heroku have one
• we can compare numbers between App hosted in US vs App hosted in Singapore
• It will verify our assumption that mounting database in Singapore is yielding result
• or if app deployed in Singpore creates a US slump for us!

[NakulManchanda]

Document proposed solution (inter-regional replication)

In addition to replication: We should also research on

• routing solution
• release & deployment strategy

eg. Backend server which communicate with DB - it should be deployed in US or it should be colocated with Singapore app etc? After we figure out replication,

• using heroku private spaces will same web server be able to connect & query to replicated DB?
• or we planning to mount local web server in same region - for read only site?

[CowMuon]

Let's get going on this, very little runway here.

[NakulManchanda]

It narrows down to two things:

• Routing

• Replication

• We have already setup new app in frack-eu, need to play-around with routing at CDN level setup

• we might need to setup reverse proxy somewhere

• Ideal routing strategy for us:

  Re-route all GET request based on geo-location of user
  POST will still hit master

[NakulManchanda]

Two options to explore:

• Cloudflare Load Balancer -> Geo Steering
  • geo location based routing only
• Clouflare Workers - custom reverse proxy to do more nuanced routing all GET based on Geolocation all POST to master