kong sync

[erosson]

from #164. we have a few options for a server - we have money to rent one now; someone's offered to host one. what we can't do is use third-parties that require their own login system, like dropbox/google.

[erosson]

4 main contenders here, I think: aws cognito, custom server app + s3, custom server app + database, or parse.com

• parse.com: tap tap infinity's used them successfully for sync with user/password. we might actually fit in their free tier still?! not sure how they handle kong's auth system. api smells very proprietary, but maybe it's just because aws is so much more familiar to me. https://parse.com/plans
• cognito's built for our use case, says their front page - game syncing with someone else's auth system. no free tier, more expensive than s3 but requires no server. unfamiliar. https://aws.amazon.com/cognito/pricing/
• s3's familiar. works. verified we cannot rate limit s3 uploads by ip, so this requires a custom server app for signed uploads, but we've done that before; or just do uploads through the server since our files are tiny this time. 1/3 the price of cognito (before adding the server). need to pick a scalable host for the server app (we are at a point where we actually have to care about scale) - ec2? heroku? appengine? google's cloud thing? maybe aws has an appengine-alike by now? I don't have a go-to place for this yet. https://aws.amazon.com/s3/pricing/
• custom app and database. database is more easily selected/sorted than s3, which may help analytics if we add extra non-encoded data to saves. possibly less scalable, but I don't think we're scaling anywhere near postgres-breaking levels just yet.

(haven't heard back from the guy who offered his server. guessing it fell through after dropbox release.)

leaning toward parse or cognito. so much simpler to avoid building a serverside app. they're both proprietary, moreso than a serverside app - but multistorage should do pretty well for migrating later if necessary.

[erosson]

we're on track for 6 million session per month. that's 12 million syncs, assuming 1 sync onload, 1 sync onclose (with no periodic syncing for crash recovery), $15 per million = $180/month. ouch. https://aws.amazon.com/cognito/pricing/ we still come out ahead but... well, that's a lot. (data storage cost is trivial. we are too big for the free tier.)

same math puts s3 at $60 for puts, $5ish for gets assuming each sync is 1 get + 1 put. that's probably a bad assumption though - the onload-sync probably won't need to put, so $30 for puts, $5ish for gets. that's dramatically better. https://aws.amazon.com/s3/pricing/ (bandwidth cost is trivial. data storage cost is even more trivial. we are too big for the free tier.) capacity for the custom app server is harder to estimate, and much more expensive.

https://www.parse.com/plans/faq parse's pricing is trickier - it's not based on requests per month, but max requests per minute - so, around burst traffic. free for 30 requests/sec, $100 for 40/sec, $200 for $50/sec, prorated - really steep price curve after we reach nonfree. we have to pick this number ahead of time, they won't autoscale it (can we adjust it via api?), and if we exceed it they throw out the excess requests which is really bloody horrible and we want to avoid that. our afternoon peak is roughly (10k sessions/hr) / 3600 seconds/hr * 3 requests per session (2 gets, 1 put; see s3 discussion) = roughly 10 requests per second at peak - we're safely within the free tier right now. is this realistic? I think so - we have lots of users but each user needs very few requests. we'll have trouble if we grow much more, though - we've about maxed out kong growth, but we're going to submit this elsewhere. yet, I think growth still works out cheaper than cognito, which needs $180 for the same growth. wow. thing is, if we misjudge our max burst, we'll start dropping syncs, and that's really bad; cognito doesn't have that problem.

man, I want to like cognito because it has the most predictible costs and it's the closest fit for my use case, but it's just so much more expensive than the other choices.

aws dynamodb is another possibility - its pricing is similar to parse's, we must reserve capacity ahead of time - but the curve for getting big isn't as steep. each write requires 4 write-units (4k), read requires 2 read-units (maybe 1, eventually consistent reads should work for us) - (10k sessions/hr) / 3600 seconds/hr * 4 writes/session = 12 writes/second, 12 reads/second = like $20, and is even within the free tier. liking this, but write/read units are pretty opaque - it's possible we missed something. also a little worried about someone using our aws credentials for their own thing, like with s3. (really, parse would have the same problem.)

[erosson]

scratch all that - since Kongregate is not an openid provider, all solutions without a custom server are vulnerable to someone dumping arbitrary amounts of data into my storage. dang!

new plan: use s3 with signed uploads.

• host an auth server which:
  • accepts a kongregate userid and auth token
  • calls kong to verify it's real
  • returns the caller a policy that allows them to write/read up to 4k to {{bucket}}/{{userid}}_{{authtoken}}.json. it can last a while before expiring.

users rarely need to call the auth server for a new policy - like, once every few days; the policy can survive through a session - so we scale really well. (will have to scale with bursty traffic, though - like at release.) after that, all sync calls go straight to s3, no dependence on the auth server.

sucks that we need a custom server after all, but after researching this the only way to avoid it is to basically allow anonymous uploads of any size/rate, or manage user accounts ourself (don't wanna do that; tti/parse do it but it's too much overhead imo).

[erosson]

https://github.com/erosson/swarm-server

aws elastic beanstalk is pretty nifty - simple appengine-like iface, with aws infrastructure. looks like it runs $35/mo or so when idle - a little over half of that is the load balancer, $15 for a micro ec2 instance. autoscales. running with 4 instances, it'd be $80/mo, but usually we'll be closer to 1. s3's extra, of course, as calculated above. in any case, costs seem reasonable.

this now works! kongregate users can remote-sync when signed in! we have an auth server! s3 credentials are cached! it's not flawless - we still need:

• [x] some visibility into cached s3 credentials
• [x] a way to force-refresh s3 credentials
• [x] autorefresh of s3 credentials once expired (should work - test!)
• [x] better error handling
• [ ] loading indicators
• [x] a way to indicate browser support - s3 relies on form-less multipart posts (the "multipart" part is important) using FormData, which IE8/IE9 don't support - but they're less than 0.1% of my users, says analytics.
  • because ie8 doesn't even work, lolol. bumped the "upgrade" prompt from ie<=6 to ie<=8.

the first three block release, but the others can wait if needed.

[erosson]

I think this is ready!