Create scaling strategy for data v1

[brishin]

Possible strategy:

• [ ] Set up nginx rate limiting
• [ ] Move server to 2nd gen EC2 mid/large instance
• [ ] Move DB to SSD

[rbtying]

Lol...

Well before that --

1. cache api key lookup (1 less SELECT call per api call) in memcached

Postgres can scale to far beyond our current usage without a slave... we should probably separate the database and the workers better.

-- Robert Ying

On 01/30/2014 02:59 PM, Brian Donghee Shin wrote:

Possible strategy:

• [] read slaves for Postgres
• [] Set up elastic IP so that web workers can scale
• [] Set up nginx rate limiting

— Reply to this email directly or view it on GitHub https://github.com/adicu/data.adicu.com/issues/69.

Robert Ying

B.S. Candidate in Computer Science Undergraduate Researcher, Columbia University Robotics Group Engineering Student Council Academic Affairs Representative Teaching Assistant, COMS W3157 Residential Advisor, John Jay 6 Phone: (408) 982-7485 (cell) Email: rbtying@aeturnalus.com

[brishin]

Do we have memcached up though on the server?

[rbtying]

No, but we should -- add it to the puppet config?

[brishin]

Hmm hmm. I don't really know puppet.

Q: Is memcached better than Redis for this? We'll be using redis in v2, so we should deploy that before I think?

[rbtying]

Memcached and redis are about the same for this, so reduce dependencies and go redis early?

On 01/30/2014 03:21 PM, Brian Donghee Shin wrote:

Hmm hmm. I don't really know puppet.

Q: Is memcached better than Redis for this? We'll be using redis in v2, so we should deploy that before I think?

— Reply to this email directly or view it on GitHub https://github.com/adicu/data.adicu.com/issues/69#issuecomment-33728915.

Robert Ying

B.S. Candidate in Computer Science Undergraduate Researcher, Columbia University Robotics Group Engineering Student Council Academic Affairs Representative Teaching Assistant, COMS W3157 Residential Advisor, John Jay 6 Phone: (408) 982-7485 (cell) Email: rbtying@aeturnalus.com

[natebrennand]

It's set to be Redis at this point so we'll stay on that track. The plan was also to put the rate-limiting in Redis, is there an advantage of doing it in Nginx? Obviously it'd be faster but can we rate limit by token that way?

[brishin]

I know we should rate limit through redis but that would require adding a bunch of code to data in time which I don't think is feasible so rate limiting mildly by IP might be a good second best option.

[natebrennand]

Oh, sorry, confusion is occurring because I used the wrong version # when I started the branch. nginx sounds like the way to go for short term then

[brishin]

Updated strategy. With nginx rate limiting, I think we should be fine on load.

[brishin]

Dammit. Rate limiting via nginx isn't possible because the actual data web layer is running on heroku if I'm not mistaken, with only the database running on the EC2 instance.