Service Breakout/Self-Serve

[bwarren2]

Per Kit's observations, we might save some money and increase stability by self-hosting some of our constellation of services. In particular, Redis has been failing in bad ways that break important things:

The contents of the full traceback was:

Traceback (most recent call last):

File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 253, in trace_task
  I, R, state, retval = on_error(task_request, exc, uuid)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 201, in on_error
  R = I.handle_error_state(task, eager=eager)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 85, in handle_error_state
  }[self.state](task, store_errors=store_errors)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 118, in handle_failure
  req.id, exc, einfo.traceback, request=req,
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 121, in mark_as_failure
  traceback=traceback, request=request)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 259, in store_result
  request=request, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 493, in _store_result
  self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 161, in set
  return self.ensure(self._set, (key, value), **retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 150, in ensure
  **retry_policy
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time
  return fun(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 170, in _set
  pipe.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute
  return execute(conn, stack, raise_on_error)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction
  connection.send_packed_command(all_cmds)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
  self.connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 442, in connect
  raise ConnectionError(self._error_message(e))
ConnectionError: Error 111 connecting to tarpon.redistogo.com:9769. Connection refused.```

And CloudAMQP is pretty expensive at $100/mo. Docker + digitalocean?

[bwarren2]

Kit has done some preliminary looking-into of this.

Suspicion: it will be easier to cut over to self-hosted things without a large user base, but it might not be worth the time and effort until we have a user base.

[bwarren2]

When we have this problem, we will be happy. If we have to do 5min maintenance at 2AM, fine.