Figure out why tasks have begun to fail

[Ravenstine]

AC:

• [x] Both ECS tasks are back online
• [x] Determine the most efficient FBIA and Google AMP support solution

[Ravenstine]

Some notes about what's happened so far:

• Initially, I tried restarting the tasks(starting the db task first). It would remain PENDING for a while and then immediately die after a few minutes.
• To diagnose what's going on, I tried to SSH into the EC2 instance. No matter what, I'd get a timeout. No connection at all.
• I restarted the EC2 instance.
• Then I tried again to start the tasks. This time they succeeded and started functioning normally.
• Still can't SSH like I used to.

It seems more likely that something went wrong at the host level rather than the container/code level. If the database storage volume had filled up then CouchDB wouldn't have been able to continue working normally. Because I can't SSH, I can't see if there are any system logs to tell me what happened. I'm going to close this ticket if I can't find a cause by the end of Friday, and then look into moving from ECS to Rancher.

[Ravenstine]

Haven't nailed anything down, although SSH is back for some mysterious reason. For the time being, I've added a CloudWatch alarm to the EC2 instance so we can be alerted when the CPU usage falls below a set threshold.