If Redis can't be contacted, the Hub stops serving content and goes down

[Splaktar]

Redis disconnect issues appear to have taken down the hub at 6:56pm ET today:

[30384] Express server listening on port 3000 in production mode
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: Redis connection to 130.211.142.195:6379 failed - connect ECONNREFUSED
    at RedisClient.on_error (/opt/hub/node_modules/redis/index.js:196:24)
    at Socket.<anonymous> (/opt/hub/node_modules/redis/index.js:106:14)
    at Socket.emit (events.js:107:17)
    at net.js:459:14
    at process._tickCallback (node.js:355:11)
[17884] worker 30384 died. Code: 1, Signal: null
[17884] worker 30389 born.

[Splaktar]

This is affecting both nodes.

Restarting Hub nodes and Redis nodes did not resolve this. Networking hasn't been changed and settings still appear to be valid.

[Splaktar]

Back online. I had to redeploy a new Redis cluster in the gdg-x network and then point the Hub nodes to this new cluster.

The previous cluster was in the default network, but there were network firewall rules that should have made it reachable. There was a warning in the redis coordinator VM that the Debian 7 image it was using was deprecated, but that should not have caused this.

[Splaktar]

We should probably update the code so that if the connection to Redis fails, it does not bring down the server.

[Splaktar]

There were issues communicating with redis-vm-3 this week. It turns out that the redis-server service on that node had stopped and didn't restart on reboot. I've reinstalled the redis-server via apt-get and it seems to be working again.

Hub is now pointing at redis-vm-3 and Hub-backup at redis-vm-1.

[Splaktar]

This is no longer relevant as we plan to remove Redis as part of #104.