search

remind101 / empire

A PaaS built on top of Amazon EC2 Container Service (ECS)
BSD 2-Clause "Simplified" License
2.69k stars 159 forks source link

emp can't control application anymore due to cloudformation timeout #1133

Closed kroselle closed 6 years ago

kroselle commented 6 years ago

I have an app already deployed for a year, it is currently up and running. I recently tried to scale it down, but empire is timing out and not giving me enough feedback to narrow down where exactly the problem is with my current knowledge of emp.

  1. emp scale -a myapp web=0:1x error: TemplateValidationError: Template URL: https://[redacted] Template Size: 4400 bytes Error: RequestError: send request failed caused by: Post https://cloudformation.[redacted].amazonaws.com/: dial tcp [aws_internal_ip]:443: i/o timeout
  2. The template does seem to exist in s3
  3. emp version 0.13.0

It seems like it is trying to get to some ip internal in the AWS side, I am assuming some EC2 instance. I searched our EC2 instances and none of them have that ip, we definitely do have subnets that would cover that ip range though. It almost seems like a machine deployed by empire died (one of the minions?) and possibly got redeployed with a new ip, but the empire db wasn't updated.

kroselle commented 6 years ago

This is resolved, we had security changes that prevented the controller from reaching cloudformation.